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ABSTRACT 


Experience has shown 


tha t the quality of 


instruction 



depends on the occupational competency of the teacher- The overall 
purpose of this project is to develop occupational competency 
examinations for nationwide use in evaluating trade and industrial 
education teachers. This first phase of the two-phase project 
involves: (1) surveying the state of the art, (2) preparing a 

handbook with guidelines and basic principles, (3) developing and 
pilot testing written and performance tests in 

elect tonics- comnunica tions and the machine trades, and (4) developing 
the structure for a consortium of states to implement the program. 
Test validity and reliability will be established in the second 
phase. The survey of the state of the art shows excessive duplication 
ct effort, lack of coordination, and deficient data, all of which 
point out the need for a national consortium. Lists of resource 
mate 1 and organizations are appended. (BH) 




£0051373 



<■; ».;- c .v. ',v : > ’ 



'.t ■ 1 ; 'V ," i. 






NATIONAL OCCUPATIONAL COMPETENCY 
7/ TESTING PROJECT 



JAN 2 2 W71 









VOLUME 2 



PHASE ONE. 

Organizing, Planning, Pilot Testing 
. — = ; . jod 

Establishing the Feasibility of a 
Consordum Concept : 



». ' ; l > ■: ■; 



Raport on 



, THE STATE OF THE ART OF ' ^ 
OCCUPATIONAL COMPETENCY TESTING, 



W»; •*' i ■' T ‘ ’ v 1 V ■ 

■ A ‘ : J 

V ;, ’i $ 7- ’-r'l 

: v : ■ 



^v/ry^- 

^V. v j *■ 1 7 > 
\-7 v v- 

'n : 7.'- 

$777' • >' 
W& i - 

^:«v. 

7; 



Adolf Panitz 



; ' C Sonias Olivo 




>; t 7 Research Project Gr« : No. 8-0474 
*\ 7/. 7* : .:'/vvi l ; ; • v v t*.: to > 

■ v Oofwrtmint of Vocational Tachrdcal Education 

*7; ;V: : . *rA7v‘ Graduate School of Education 3 - 7* V'i'; 

, flutort, TN State UnHrWty, Now 8mn*vrick Ntw Jersey 

\ ;V ®V ** U. S. Uapoftrtront of Heelth, Ediwadon and Welfare 
,'r 7 yH'". Office of Education ££':*■ ^ 

Buraau of R match 



,f - 1 i > 

i/r 




ED051378 



NATIONAL OCCUPATIONAL COMPETENCY TESTING PROJECT 



A Consortium for Occupational Competency Testing 

Of 

Tradn and Industrial Technical Teachers 



PHASE ONE 

(Organisation, Planning, Pilot Testing) 

US DEPARTMENT OF HEALTH EDUCATION 
k WELFARE 

OFFICE OF EDUCATION 

THIi DOCUMENT HAS BEEN REPRODUCED 

Exactly as received from "he person or 
ORGAN ZATION ORIGINATING I’ POINlS OF 
VIEW CR OPINIONS STATED OL NO' N EC EG 

sar/ey represent omc.'AL of. c t or euu 

CATION POSITION OR TOUCY 

A Report on 

THE STATE OF THE ART OF OCCUP ATI ONAL COMPET CNCY TES TING 

By 

Adolf Panitz and G. Thomas Olivo 



Research Project Grant S-0474 to 
Hie Department of Vocational- Technical Education 
Graduate Tchool of Education 
Rutgers, The State University, New Brunswick, N.J. 



June K, 1970 



■■ ^ 

The research reported herein was performed pursuant 

to i> grant with the Office of Education, U S. Depart- 
ment oi Health, Education, and Welfare. Contractors 
undertaking such projects under Government sponsor* 
ship are encouraged to express freely their profes- 
sional judgment in the conduct of the project. Points 
of view or opinions stated do not, therefore, necessar- 
ily represent official Office of Education position or 
policy. 




United States Department of 
Health, Education, and Welfare 

Office of iuucation 
Bureau of Ret^arch 



NATIONAL OCCUPATIONAL COMPETENCY TESTING PROJECT 



PLANNING COMMITTEE 

Dr. C. Thomas Olivo, Frcject Director 
Dr. Adolf Panitz, Associate Director 



PRINCIPAL INVESTIGATORS 



Dr. Carl Schaefer, Chairman 
Department of Vocational Education 
Graduate School of Education 
Rutgers University, New Jersey 

Dr. Melvin Barlcw, Director 
Division of Vocational Education 
Graduate School of Education 
University of Southern California 
Lor Angeles, California 

Dr, Richard Nelson, Chief 
Bureau of [ndustrial Education 
California State Education Department 
Sacramento, California 



REPRESENTATIVE LEADERS 

Walter Bialobrzeski - Connecticut 

Dr. Calvin J. Cotrell - Ohio (Research Center) 

Dr. Edward Kankin - Jamaica, West Indier 

Ernest Kramer - Washington 

Dr. Gordon McMahon - New York 

Dt*. Robert M. Reese - Ohio 

Dr. Byrl Shoemaker - Ohio 

Dr. Merle Strong - Wisconsin 



EX-OFFICIO 



Dr, Lance Kodes Washington, D. C. (Project Officer) 

Dr. Lawrence Braaten - Washington, D. C. (Resource Consultant) 




i 



:<r 



FOREWORD 



For an industrial nation which constantly accele/ates its technolog- 
ical skills and knowledge, the rate of growth depends upon the productive 
potential of the av .lable labor force. Leaders of the industrial sector of 
the nation, and those concerned with utilization of the products and pro- 
cesses of industry, racognize that the socioeconomic well being cf the pecple, 
their health and tne national security are inseparable from manpower develop- 
ment potential. 

For some, this means preparation for initial employment. Others, in 
the labor force, may require upgrading or extension training, so that they 
may perform at levels requiring higher skills and technical knowledge or 
serve middle management as foremen, supervisors or managers. Those who may 
be displaced, or need tc prepare for employment in other occupational fields 5 
require retraining. 

Whether vocational education relates to the preparation of youth, new 
workers, or the retraining or upgrading of adrlus; whether the training is 
carried on in educational institutions, industry or the military, there is a 
force, common to ^11 to make the potential manpower actual manpower, and that 
force is the occupational competence of those who serve as teachers. 

Within the United States and other industrialized nations, the demand 
foi highly skilled craftsmen and technicians to prepare themselves for teach- 
ing is rapidly growing. Unlike other teachers, those who serve in trade and 
industrial education and training develop their subject competencies in. actual 
live situation through full-time "hands on 1 ' employment. 

Experience has shown that the quality of instruction, whether for a 
beginning learner or advanced craftsman., depends directly upon the occupational 
competency of the teacher. Thus, the need for instruments for the evaluation 
of the practical skills and related technical competency of potential teachers 
becomes obvious. 

Phase One, of this project, sought to establish the feasibility of pool- 
ing national resources into a Consortium of States for occupational competency 
testing, and to develop and apply the most effective methodology to the prepa- 
ration, administration, pilot testing, evaluation and reporting of two occupa- 
tional competency tests in two major trada and industrial/technical occupations 

For 2 atarting point, it became necessary to establish certain ’’bench 
marks 11 . This part of the project relates to people and organizations involved 
in developing adequate measures for occupational competency. Their experiences 
and the results of their efforts were carefully reviewed and are here recorded 
as Me Report on "The State of the Art" of occupational competency testing. 
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Acknowledgment is made to those who assisted in this massive under- 
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on this project. 
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This report consists of four parts. Fart One provides the background, 
rationale and resources of the study. Part Two includes the research studies 
and literature relating to occupational competency testing. Th? experiences 
with occupational testing materials and practices throughout the United States 
and the Territories are reported in Part Three, Occupational testing materials 
and programs of industrial, governmental, labor and military organizations are 
reviewed in Part Four. A summary of the findings of occupational competency 
and recoamendat ions resulting from the study are presented in Part Five, Four 
Appendices of significant resource materials, which were examined, complement 
this report. 



The conclusions are those of the authors. The contributions to vocational 
education which may result from this report wa gladly share with the many who 
have so wholeheartedly cooperated, and offered support when it was most needed. 
However, the responsibility for the point of view expressed and for any errors 
of omission of commission which may have occurred, is ours, 

C. Thomas Olivo 
Project Director 
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PART ONE 



BACKGROUND , RATIONALE AND RESOURCES FOR THE STUDY 

I. OCCUPATIONAL COMPETENCE: THE KEYSTONE OF QUALITY TEACHING 

The level of occupational competence of the teacher, of trade 
and industrial education, is the keystone upon which the whole instruc- 
tional program is built. This position recognizes that the prospective 
teach must also possess personal qualities, professional preparation in 
a quality teacher training program, and other qualifications which col- 
lectively insure success in teaching. Foundat ionally , there must be 
evidence that the range of occupational competence is adequate to meet 
the full spectrum of needs of in- school and out-of- school youth and 
adults. This occupational competence is essential for teaching in pub- 
lic and private schools, industry, governmental agencies, the military; 
whenever skills are to be developed and the related technology mastered 
for a trade, industrial service or industrial technical occupation. 

A few states I.:ve been able, with varying degrees of success, 
to develop or participate in establishing the occupational competence 
of prospective teachers. The development, admin is trat ion and valida- 
tion, of occupational competency tests requires human and material re- 
sources and finances that are in short supply even in leading and 
heavily populated industrial states. At the same time, all states rec- 
ognize the earlier stated truism that instructional excellence must be 
rooted in the occupational competence of each teacher. 

Many state officials have stated frankly that their departments 
are not equipped to develop new examinations, keep old ones up to date, 
or even develop a single trade or industrial test. There is agreement 
that the developing, printing, distribution, adminis tra t ion , scoring and 
validation of occupational competency tests imposes an unrealistic burden 
and that no single state has adequate resources of dollars or experienced 
personnel to resolve the problem. 



II. STEPS LEADING TO THE ACCEPTANCE OF THE CONSORTIUM CONCEPT 

Two one-day institutes at Rutgers University (1966), attended by 
repr • scr^ ativcs of twenty- three (23) states, concluded that the develop- 
ment and implementation of an occupational competency examinations pro- 
gram on a nationwide basis would be a more efficient use of personnel and 
would provide higher quality examinat ' ons . Almost unanimous agreement was 
reported that such examinations would be used.l 

These same states and ten additional ones who met in Chicago on 
March 13-14, 1970, except for two abstentions, stated unanimously their 
endorsement of tho. concept of a consortium of states arrangement, and 
their willingness to participate in a National Occupational Competency 
Testing Program. 



ERIC 



11 



Everything "he staff researched and every person and agency con- 
tacted revealed the common problems: The military with their highly spe- 

cialized tests, industry, educational testing, services, labor--all rein- 
forced the basic facts of life 

(a) that occupational competency was a prime requirement 
for the teacher, 

(b) the instruments and techniques each was using for 
measuring occupational competency of teachers of 
trades and industry were not as reliable, valid 
or practical as each indicated they should be, 

(c) no single agency had all the essential resources 
of hardware, knov’ledge, finance and skilled test- 
ing personnel, 

(d) their efficiency and effectiveness in occupational 
competency testing required the cooperative pooling 
of human and material resources, and 

(e) the national consortium concept advanced a plan 
worthy of endorsement and implementation. 

So, the time was right to secure, review, assess present prac- 
tices and report the findings, and establish a point of reference for the 
whole movement. In essence, this was the purpose for this intensive sur- 
vey and study of the State-of-the-Art in occupational competency testing, 



III. OVERVIEW AND OBJECTIVES OF PHASE ONE 

The present project includes two phases. Each phase involves 
research, test preparation, pilot testing, the gathering and analysis of 
data relating to the tests and testing and the establishing of principles 
of operation and procedures essential for the development, organization 
and administration of a consortium of states for a National Occupational 
Competency Testing Program, 

The outcomes of Phase One should establish the feasibility of 

(1) developing trade and industry teacher competency 
examinations acceptable to the many states by ac- 
tually developing and piloting such tests, and 

(2) laying the groundwork on which to proceed if the 
states accept the consortium concept. Phase Two 
would relate to establishing validity and reli- 
ability by in-depth testing, developing a signif- 
icant number of trade and industrial competency 
tests to assure success in a consortium arrange- 
ment and to make the total plan operational. 
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A. Scope of Phase One 



Phase One involved: 

(1) A survey of the state-of-the-art of occu- 
pational competency testing of teachers of 
trades and industry tu determine: 

a. the extent to which occupational compe- 
tency tests are being used in the various 
states , 

b. the practices and procedures of occupa- 
tional competency testing employed by 
industry and/or labor organizations, 

c. the methods and procedures of occupa- 
tional competency testing used by the 
military, 

d. the methods and procedures of competency 
appraisal used by government agencies 
(civil ser\ice), 

e. the availability of tests prepared by 
private research organizations, 

(2) The establishment of basic principles and Lhe 
preparation of guidelines for the development 
and administration of written and performance 
tests for qualifying prospective vocational 
industrial teachers. The material resulting 
from this wc rk will be printed \s a Handbook 
on Occupational Compe 1 ency Testing. 

(3) The development, reproduction and pilot test- 
ing of written and performance tests for the 
Electronics Industries Occupation (Communica- 
tions) and the Machine Industries Occupations 
(Machine Trades), 

(A) The defining of guiding principles, policies, 
and a basic operating structure for the estab- 
lishment of a consortium of states, the bring- 
ing together of interested states to explore 
their interest and to establish a base for im- 
plementing the concept. 
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IV, THE NEED FOR A SURVEY OF IKE STATE-OF-THE-ART 



A number of states- -Cal ifornia, Florida, Michigan, New York and 
Pennsylvania, have introduced competency examinations for admission to 
vocational industrial technical teacher education programs, for certifi- 
cation, and for credit for undergraduate and graduate programs of study. 

A survey by Schaefer, conducted in 1959, showed that 16 states were then 
using tests to evaluate occupational competency. ^ in a more recent study, 
by Kazanas and Kieft,^ involving a review of State Plans, it was found 
that 11 states used trade competency t :sts to help certify vocational ed- 
ucation teachers. In the same study, 8 state plans provided for the use 
of occupational competency tests; 5 other states use competency tests, 
although no special provisions were included in their state plans for such 
purpose. Editorially, it should be stated that members of the Planning 
Committee indicated that many other states require the satisfactory pass- 
ing of an occupational competency examination for entrance into the teacher 
training program, even though such detail may not appear in the state plans 

Recent discussions with officials responsible for vocational edu- 
cation and teacher certifications in several of the states revealed that 
considerable dissatisfaction exists with the quality of the tests now in 
use. These tests arc largely of the paper and pencil variety with some 
requiring actual performance in simulated job situations. There is little 
evidence to indicate that test questions were pretested, or that the instru 
ments were validated according to acceptable test development procedures. 



V. CONCERNS OF OTHER RESPONSIBLE AGENCIES 

Evaluating occupational competence of tradesmen and technicians 
is not necessarily confined to future vocational teachers. Industrial 
employers, government agencies, the military, and labor o' ganizations are 
also faced with evaluating the occupational competency of individuals for 
adequate performance in their jobs. Since each organization has a similar 
problem of occupational competency evaluation it is reasonable to assume 
that there organizations have done work in this area. Therefore, a survey 
of the "State-of- the- Art 11 must extend beyond the practices of the various 
vocational education units in the states and include a review of existing 
test procedures in somewhat related areas, rield visits and observations, 
as well as a search of published related materials. 



VI. IDENTIFICATION OF RESOURCES 

The first step in the " survey-of- the-art n of occupational compe- 
tency testing involved the locating and collecting of as much relevant 
information as possible. It was recognized that such effort involved a 
multiple approach which included a review of the literature competency 
testing, field visits and observation of existing practices, as well as 
personal and/or mail contact with State Directors of Vocational Education 
and other professional personnel in each state, the District of Columbia; 
Virgin Islands and Puerto Pico; Goveriiment Agencies (Federal, State and 
Local); Civil Service Agencies; the Armed Forces; Private Research Organi- 
zations; Business; Industry; and Labor Organizations. 
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Additional contacts were made through professional organiza- 
tions, such as: National Association of State Directors, the National 

Association of Industrial and Technical Teacher Educators, the Research 
Committee of 'he Trade and Industrial Education Division, American Voca- 
tional Association; and the National Vocational Education Research Centers 
at Ohio State University and North Carolina State University; the Univer- 
sity of Wisconsin Center; the various State Vocational Education Research 
Coordinating Unit s , and the American Psychological Research Association. 

Referral to other resources were made by the American Manage- 
ment Association, the National Association of Manufacturers, A? "ociat ed 
Electrical Industries; National Tool, Die and Precision Machining Asso- 
ciation; American Association of Industrial Management, and the Industrial 
Conference Board. Private research and contract organizations involved in 
the field of testing, such as: the Psychological Corporation, Educational 

Testing Service, Science Research Association, were also contacted. 

Among organized labor, the National Labor Organizations, who 
carry on some form of occupational competence evaluation for plumbers, 
carpenters, machinists, and electrical craftsmen, were contacted. 



PART TWO 



RESEARCH STUDIES AND LITERATURE RELATING 
TO OCCUPATIONAL COMPETENCY TESTING 



I. SOURCES AND NATURE OF OCCUPATIONAL COMPETENCY TESTING AND RELATED 
RESOURCE STUDIES 

The review of literature search and analysis centered around 
ail printed reports on occupational competency tests. Work was carried 
on in the libraries of Columbia University, New York City Central 
Library, New Rochelle Library, the Library of the New York State Edu- 
cation Department and the Library of Congress, 

Additional research work was done at the National Center for 
Vocational-Technical Education, the Ohio State University; and through 
ERIC* Abstracts of research and related materials for vocational- tech- 
nical education for the period 1967-69 were reviewed in the ERIC library 
for contributing experiences and related irt onna tion , This search, in 
the specialized areas, produced only four studies somewhat related to 
occupational competency testing but primarily concerned with teaching 
competence instructors in vocational education,^ One study related 
directly to occupational competence. ^ 

A search at the Wisconsin Center for Studies in Vocational and 
Technical Education revealed the same studies as Ohio. 

In the libraries, the catalogs and abstracts, pertaining to 
testing, and tba general catalogs were checked for materials on occu- 
pational competency testing. The search was extended to include peri- 
odicals, magazines and special publications of the American Psycholog- 
ical Association and the American Psychological Research Association. 

In addition, textbooks on testing and occupational testing were reviewed. 
Only two volumes were found that concerned themselves with construction, 
administration, evaluation and validation. ^ One of these publications 
is cut of print. ^ 

A . Occupational Competency Tests Based On Trade Analyse s 

Evaluation of occupational competency is becoming increas- 
ingly important for teacher certification. Kazanas and Kieft^ concluded 
i-hati 

(1) well designed examinations have value in teachei 
certification, and 

(2) examinations can be effective in determining 
teacher competency, increasing the number of 
teachers, and in indicating weaknesses of those 
who fall. 
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Their conclusions were based on oral, written and performance 
examinations which were developed from several trade analyses combined 
for each trade area to form a comprehensive analysis. This analysis 
was revised by the research staff and teacher educators. 

Approximately, 1500 test items were prepared for each trade 
area. These were screened, evaluated and divided into three separate 
forms for each trade. The committee v?ho evaluated the written exami- 
nations also made recommendations concerning performance test content 
and developed a list of practical tests for the evaluation of manipu- 
lative skills. As of th _s date, validity, reliability, and general 
usefulness have not been established. This work is expected to be com- 
pleted in Phase II of th e i v project, 

B. Evaluation Of Teaching 

Popham, in his final report: ,T Perfomance Tests of Irst rue- 

tor Competence for Trade and Industrial Education 1 ' ,9 described teaching 
proficiency tests for automotive mechanics (carburat ion) and for elec- 
tronics (power supplies). An assessment was made of each test's ability 
to distinquish between the experienced teacher and the non- teacher with 
respect to his ability to achieve pre- specif ied instructional objectives. 

All participants were given sets of operational defined ob- 
jectives. They attempted to achieve goals during an instructional period 
of 10 hours. Pre and post-cests, based explicitly on the objectives, 

Were given to the students cf each participant. Average class achieve- 
ment was used as an index of the teacher’s proiiciency. 

Comparison of student perforr ance data revealed systematic 
differences between the performance of teacher and non-teacher groups in 
autc mechanics or electronics. Results were attributed to problems asso- 
ciated with the training of teacl irs, as well as the reinforcement struc- 
tures operating when they commenced training. This project involved an 
evaluation of teaching rather than occupational competence. 

C , Impact of Educational Variables On Professional Com petencies 

In the study T1 Professional Competencies of Teachers of Tech- 
nical Education in Florida" , Kurth and Giannini^ attempted to determine 
whether the professional cornetency of technical education teachers was a 
function of several educational background variables. Teachers were 
judged on the basis of a basic comprehensive examination, a sociome :r ic 
rating of professional colleagues and the level of cooperation shown dur- 
ing the study. Among the background variables were the amount of occupa- 
tional experience other than teaching and the classification of other 
occupational experience. 
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The authors concluded that there were no significant dif- 
ferences between the college attended, time intervals between degrees 
and certification, and the number of years employed in other occupa- 
tions. It is difficult to appraise the role assigned to occupational 
competence in this study. No detailed explanation x^as given as to the 
content of the basic comprehensive test nor how experience was appraised 
other than in years of service. 

D. C ollegiate Credit For Validated Occupational Competence 

A great deal of concern is voiced about granting college- 
university credit for trade and industrial experience in vocational 
industrial- techn? cal departments, and by certification units of State 
Departments of Education. Lauda^ surveyed 201 colleges and univer- 
sities, The results showed that one-fourth of the institutions sur- 
veyed grant credit for trade and industrial experience; others planned 
to do so. Significantly, he mentioned that most institutions use a 
combination of written, oral and skill tests to evaluate competence 
before granting credit. The New York State IJniversitv College, Oswego 
grants credit when proof of competency is demonstrated through success- 
ful passing of a test. Candidates are eligible to take the test only 
after they have offered proof of the required experience. Unfortunately, 
nc explanation was offered as to the types of examinations employed, 
their validation and reliability, and the method of administrative con- 
trol. 



E . Validatin g Occupational Competence Through Trade Competency 
Tests 



New York, California and Tennsylvania have utilized written 
and performance tests for a number of years as part of the whole process 
of establishing occupation competence, as a pre-requisite to teaching in 
a trade or vocational industrial area. Koenigsberg and Reilly 12 inves- 
tigated the reliability of New York State’s written and performarce exam- 
inations; i.e, do the examinations yield scores that are relatively stable 
for any individual. Also, in terms of the validity of the written and 
performance examinations, do they reflect the areas in which the applicant 
should possess skill or knowledge (content validity), and do the examina- 
tions differentiate the applicant's degree of skill or knowledge. 

The proficiency examinations presently in use were developed 
statewide over r period of years and were essentially "teacher nadV. The 
test items were written by trade and industrial teachers using a guidebook 
that had been prepared and refined continuously by the vocational industrial 
teacher training staff as part of the Bureau of Industrial Teacher Training 
of the New York State Education Department. It was assumed that these tests 
were valid and reliable prior to the present study. The three most widely 
used examinations, Auto Mechanics, Cosmotology and Machine Trades, were 
selected for investigation and further revision. 
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It should be reported that, in the last five years, the 
industrial teacher training function was transferral from the New York 
State Education Department to newly established Divisions of Vocational 
Technical Education at State University College at Oswego and Buffalo, 
and the City University of New York. Since that time, the function of 
occupational competency development and coordination has been carried 
on at Oswego under the leadership of Dr. Cordon McMahon, Division Direc- 
tor. Funds were provided for the Koenigsberg-Reilly study by the New 
York State Education Department. 



F . Establishi n g; Test Reliability 



The final report of this study discusses the methods em- 
ployed to process examination information by computer and other data 
processing equipment. An item analysis was performed, including the 
difficulty index (the percent passing each item), and the point biserial 
coefficient (a correlation of the item with the total test score). 



For appraisal of reliability, the 
was employed which yields a measure of interna 
on a test have high intercorrelation with each 
much the same attribute, the reliability coeff 
intercorrelations are low, either because the 
attributes or because of the presence of error 
efficient will be low”, Ferguson 1966, 



Kuder- Richards on Formula 20 
1 consistency. "If the items 
o tlier and are measures of 
icient will be high. If the 
item measures different 
, then the reliability cc- 



Another estimate of the reliability of an examination em- 
ployed was the standard error of estimate. The standard error of measure- 
ment is the standard deviation of a sample of scores for an individual 
around his true score. The greater the ratio between the standard error 
of measurement and the standard deviation of the test, the more reliable 
are the test scores. Additional important reliability data, dealing with 
difficulty levels and point biserials, were processed by the authors and 
reinforced the information obtained by other methods. 



For the performance tests--the interjudge reliability of the 
exam iners-- the rating scales for each pair of judges were compared to 
determine the extent of agreement on the ratings. A high degree of corre- 
lation was obtained. This study established the feasibility of using sta- 
tistical item analysis and cross-validation procedures to assess and improve 
the effectiveness of occupational competency testing programs. The results 
of this study established the need for further revision and improvement of 
examinations. They also indicated that examinations designed by teachers 
tend to include test items known to teachers but not necessarily to prac- 
titioners in the trade or technical occupation. 
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G . Significance of Correlation BeLween Writte n And 
Performance Test Scores 



14 

Another study by Reilly was concerned with norms patterns 
applied to the selection of trade and technical teacher candidates. The 
data treated included- region of the state, age, field (occupation), IQ 
score, written occupational competency test score, performance test score. 
State University of New York examination score, years experience, course 
grades, and number of courses taken. 

The report on this study includes a profile of the trade and 
industrial teacher candidates. However, Reilly cautions that the typical 
member of any sample of applicants probably does not exist. Nevertheless, 
such a description seems useful in trying to understand the. nature of the 
candidate sample. From this study, it appears that trade knowledge (theory) 
and trade performance (practical) tests are not highly correlated, and that 
written examinations alone are a poor indicator of trade competence. 

This conclusion is supported by Stuif who carried on extensive 
work in performance measures for the Navy during World War II Stuif says, 

’’Although it had been assumed that written tests sufficed to indicate what a 
man had learned in a service school, the evidence showed that performance 
tests and improved shop grades were not closely related with written test 
grades" . 



Uditorially speaking, the project director and associate 
director support the position expressed by Dr. bdward Hankin,15A a member 
of the Planning Committee and an experienced authority on tests and measure- 
ments, particularly occupational competency testing for trade and industrial 
education teachers, in a careful review of this report: 

"...♦.that the correlations should be high if the study 
population, from which the scores are drawn, represent a 
normal range of competency from low to high. Hankin 
found, rather consistently, that if such correlations 
are low, one or the other of the tests, usually the 
written test, is weak. Conversely, merely refining a 
written test will increase substantially the correla- 
tion. 

"It might be pointed out that there is a theoretical base 
for expecting high correlation in this situation. Obis 
gives one method of interpreting a coefficient of correla- 
tion as follows: ’A coefficient of correlation may be 

thought of as the decimal fraction which tells vhat pro- 
portion of the causes affecting the magnitude of two vari- 
ables are common to both variables.' Since, in this case, 
tnere is one 'cause' (occupational training and experience) 
to measure with the two var5.ables (written and performance 
tests) and, if the scores of these tests both reflect only 
the one and the same cause, perfectly valid tests (theoret- 
ically) should produce a perfect high correlation. This, 
of course, is not practical. 
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"The notion that if there is high correlation only 
the written test would be needed derives from other 
fields of testing, such as mental ability, where the 
intention is to isolate independent traits like space 
perception and verbal reasoning* Here a low, or even 
negative correlation, 1c needed to show them as inde- 
pendent* In measuring the occupationally competent 
worker, he has both types of abilities (as measured 
by the two types of tests) , and they are not inde- 
pendent* High correlation of the examinations is no 
basis for assuming the candidate has skill and tech- 
ical competence because he has one or the c. her, 11 

T, The mere fact that much that is in the written tests 
might be mastered simply by classroom instruction and 
study would make the written test the poorest basis 
for such an assumption*' 1 

"By such means, written and performance tests, the 
breadth of competency can be checked. If only a 
total score is produced, each individual's compe- 
tencies are "averaged out 11 . Two people with the 
same score might be quite different in their compe- 
tency and there is no clue as to what their differ- 
ences are or even that they arc different. For 
example, if there were tv auto mechanics, one might 
have limited experience or none on engine tune up 
and the other none on chassis work* r ihus, the sub- 
test scores are valuable also for their diagnostic 
value in establishing the candidate's capability 
across many major parts of a trade*" 

Reilly^ touches on another problem presently under serious 
discussion; name ly, the number of years of experience essen lal for teach- 
ing in vocational industrial education and training programs. In this 
study, the correlation between five years of journeyman experience and all 
other variables was negligible. All candiates had five years or more years 
experience at the journeyman level, Reilly concludes that occupational 
experience beyond five years, at the journeyman level, is not important. 
This conclusion could be challenged by the limited scope of the study. In 
view of the importance of this issue and the difficulty in some states of 
recruiting competent people with appropriate experience, further research 
on the number ot years of experience required is essential. Part of the 
problem may relate to salary structure in comparison to industrial wages. 



II, OTHER APPROACHES TO MEASURING ACHIEVEMENT 

In a final report concerned with the Development of a Model 
Vocational Teacher Education Program, Miller^ proposes an occupational 
testing program somewhat different from those in other states. A compe- 
tency test is proposed for measuring, gaging and layout work which a 
trade and industrial teacher should be able to demon s t rat o ( supplemented 
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by another list containing 136 items in the area of machining. The 
report does not indicate that written and performance examinations 
have been developed, nor is there any indication how such tests are 
to be validated. 

A still different approach to measure student achievement 
is being developed by Baldwin ^ . Thi: extensive study is primarily 
concerned with the level of achievement attained by students in post 
high school trade and technical programs. Emphasis is placed upon a 
conceptual framework for classifying the behavior to be measurr . in 
each of eight curricula. These tests are being validated on student 
populations after one year of training for segments of the various 
trades. At this point, it is doubtful whether the results will be 
applicable to the competency evaluation of future trade and technical 
teacher s . 



III. STATE EFFORT AT EXTENSIVE TESTING TO 
ESTABLISH RELIABILITY AND VALIDITY 

California haa carried on occupational competency testing 
for a number of years. In a brief paper, Lofgren describes the 
efforts made and the procedures employed. For illustration purposes 
only, he has selected the auto mechanics competency examination to 
describe item analysis, measures employed to establish reliability 
and the factors involved to validate the test. Two measures of test 
reliability were used. The primary reliability coefficient (Kuder- 
Richardscn) is based on the core ip t of "rational equivalence" which 
stresses the consistency of the subjects responses to all items on 
the test and, thus, provides a measure of the internal consistency of 
the test. 



The second measure (Gutman) is a split half-index which 
estimates total test reliability by subdividing test items into two 
subsets and provides an estimate of the equivalence of the two test 
scores. Both of these coefficients can be computed upon a single, 
administration of the test. This is important in the practical as- 
pect of test development. 

While there may be a question on the comparability of the 
resulting estimates on theoretical grounds, in actual practice the 
obtained difference would not be very large. If large differences 
occur, it may bo an indication that the test probably contains het- 
erogeneous items; that all parts of the test are not measuring the 
same thing. 

Hie validity of the test was analyzed to see how well it 
correlated with the criteria it was supposed to measure. Lofgren 
emphasizes the difficulty of obtaining adequate criteria. Two 
validity criteria were established. The first involved the number 
of years of work experience; the second, that of former employer 
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ratings of trie journeymen auto mechanics involved in the sample. A 
modified tetraehoric correlation formula was used in the statistical 
treatment . 

Included in this paper are the formulas; employed and statis- 
tical tables resulting from the treatment of the data. The importance 
of this paper lies in the attempt to develop occupational competency 
tests, in accordance with accepted test development procedures While 
there may be disagreement on some of the details; and t' e actual means 
employed, nevertheless, this work is a step in the direction of devel- 
oping better tests. This paper was limited to written tests and con- 
tained no information on performance tests. 

IV. EXTREMELY LIMITED RESEARCH ON OCCUPATIONAL COMPETENCY TESTING 

A greet deal of material has been published dealing with intel- 
ligence, attitude, aptitude and achievement in academic subject areas. 
Some tests have been developed to evaluate office occupations, particu- 
larly: typing, shorthand and transcriptions. For other occupations, 

little or no research has been published. Several tests in auto mechan- 
ics, machinist and lathe operation are briefly described, but nc evidence 
of validation and reliability is indicated. The ’’Mental Measurement Year- 
13 ook ,T , 3rd, 4th, 5th and 6th editions^, and Burns 1 “Tests in Print" 2 0 
provided very little additional information on the research carried on in 
occupational competency testing, or in the availability of tests through 
private organizations. 

The review of the literature confirmed the earlier belief of the 
principal investigators, study staff, planning committee, and other voca- 
tional industrial leaders of the extremely limited amount of recorded 
experiences about occupational competency testing. This is appalling, in 
view of the fact that a few states have required the satisfactory passing 
of an occupational competency examination to measure and validate occupa- 
tional experience for over 40 years. The situation is further aggravated 
when industry, government, the military, and others need to place such 
dependence on the results of valid measures of occupational competence. 

At this point, Shimberg 1 s cryptic remark is noted: 

"When one considers the Importance of performance 
evaluations to the future of vocational education, it 
seems inconceivable that so little progress has been 
mde. M 21 
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PART THREE 



OCCUPATIONAL COMPETENCY TESTING MATERIALS AND PRACTICES 
THROUGHOUT THE UNITED STATES AND TERRITORIES 



I „ APPROACH TO STATE CLEARANCES AND SOURCES OF INFORMATION 

State Directors of Vocational Education, State Supervisors 
of Trade ard Industrial Education, Head State Trades and Industries 
Teacher Educators, and individuals in charge of State Research Co- 
ordinating Units for Vocational Education were the lead persons con- 
tacted by letter ► The rationale, the vocational objectives, and scope 
of the project were explained soliciting their assistance to provide 
specific relevant information needed for this research effort. These 
requests were supplemented by a brochure entitled, "Occupational Compe- 
tency Testing". The following indicate the type of information re- 
quested from the states: 

1. A list of titles of tests which has been prepared, 

2. Titles of other tests in preparation, as either 
revisions or new tests, 

3. Plans for the future development of occupational 
competency tests, or establishing a testing program, 
or test procedures, 

4. bamples of written and practical examinations for 

a) the machine industries occupations (machine 
t rades) , 

b) electronics industries occupations 
(communication) and 

c) automotive industries occupations (automotive 
trades) , 

5, Guidelines for the construction of new tests, 

6, Instructions for examiners giving tests, 

7, Instructions for candidates taking tests, 

8, Method of evaluation of test results 

A second follow-up Letter was sent to those stales and 
institutions which did not respond at first. Table I represents the 
total number of replies received. 
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Table I - Responses to Initial Requests 
on Occupational Competency Testing Practices 




Contact Persons 


Number of 
Requests 
Sent 


Respondents 


Percent 

Returns 


1 


State Directors of 
Vocational Education 
or equivalent 


54 


40 


74 


2 


Head State Industrial 
Teacher Educators 


63 


37 


59 


3 


State Research Coordina- 
ting Units for Vocational 
Education 


46 


8* 


17* 


4 


State Supervisors of Trade 
and Industrial Education 


56 


42 


75 




TOTALS 


219 


127 





* This limited response is recorded even though it doe3 not 
portray an accurate picture. Many RCU directors turned the 
request over to an appropriate unit within the state that 
handles this function. In some instances, it was the head 
state industrial teacher educator; in others, a state 
supervisor, etc. 



II. STATE USE AND DEVELOPMENT OF OCCUPATIONAL COMPETENCY EXAMINATIONS 

In a few states, the examinations are developed and admin- 
istered by the Division of Vocational Education within the respective 
State Departments of Education. However, in the majority of states, 
this task is carried on by Departments oc Vocational Industrial Techni- 
cal Education in one or more colleges or universities, approved by the 
State Departments of Education to conduct a Trade and Industrial Teacher 
Education Program. The listing in Table II indicates the states and/or 
the centers involved in occupational competency testing who responded 
to the inquiry: 




15 






Table II - State Practices for Developing 
Occupational Competency Tests 



S tale 


Agency or Institution 
Responsible for Test 
Development 


Notes 


Cal ifornia 


Department of Education 
Bureau of Industrial 
Education, Sacramento 


A major responsibility now 
is carried on within the 
Division of Vocational 
Education, Graduate School 
of Education, UCLA, 

Los Angeles 


Colorac j 


Colorado State Univer- 
sity, Ft. Collins, 

Dept, of Vocational and 
Technical Education 




Connecticut 


Central Connecticut 
State College, 

New Britain 




Flor ida 


Florida State Univer- 
sity, School of Educa- 
tion, Department of 
Industrial Arts and 
Vocational Education, 
Tallahassee 

Florida A & M University 
School of Technology, 
Tallahassee 

University of Miami, 
Coral Gables 




Georgia 


University of Georgia 
College of Education 
Division of Vocational 
Education, Athens 




-'Hawaii 


University of Hawaii, 
Honolulu 




Indiana 


Purdue University 
Department of Vocation 
al Education, 

Lafayette 




Kansas 


Kansas State College 
Department of 
Vocational Education, 
Pittsburgh 
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"Kentucky 



Louisiana 



Massachusetts 



University of Kentucky 
Department of 
Vocational Education 

Louisiana State Univer- 
sity and Agricultural 
and Mech. Col lege , 

Baton Rouge 

State Department of 
Education, Division of 
Vocational Education, 
Boston 



Specific informat ion not 
provided as to extent of 
test use, purpose, etc. 



Michigan 



P'astern Michigan Univer- 
sity, Department of 
Industrial Education, 
Ypsilant i 



Ferris State College, 
Big Rapids 



Minnesota 



Nebraska 



New York 



Ohio 



Wayne State University 
College of Education, 
Detroit 

University of Minnesota 
Department of Industrial 
Education, 

Minneapolis 

Kearney State College, 
Kearney 

State University 
Division of Vocational- 
Technical Education 



Instructional Materials 
laboratory, Division of 
Vocational Education, 
State Department of 
Education, The Ohio 
State University, 
Columbus 



Occupational Competency 
Tests for high school and 
post-high school students 



Other Examinations are 
administered by: Division 
of Vocat ional- Technica 1 
Education, State Univer- 
sity College, Buffalo and 
Division of Vocational- 
Technical Education, City 
University of New York 

Occupational Competency 
Tests for high school and 
post-high school students. 
The Universities of Cincin- 
ati, Ohio State University, 
Toledo and Kent State use 
examinations for credit 
towards degrees In trade ind 
industry education. The use 
of tests for certification 
is permissive not mandatory. 
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Oregon 


Oregon State University, 
Division of Vocational- 
Technical Education, 

Cor va 115 s 




Pennsylvania 


Temple University, 
Philadelphia State 
College, College Park, 
University of Pittsburgh 


Occupational Competency 
Testing is coordinated as 
a function of the State 
Department of Public 
Education 


South Dakota 


Southern State College, 
Springfield 




Tennessee 


The University of 
Tennessee, College of 
Education, Department 
of Vocational-Technical 
Education, Knoxville 


This university serves as 
a testing center for a 
consortium of southern 
and central states. 


Texas 


Texas A & M University 
College of Enginee; Eng, 
College Station 





* States or territories interested and planning to develop occupational 
competency tests. 



The states and institutions shown in Table II use occupa- 
tional competency examinations for three main purposes: 

1, for admission of potential trades and industries 
teachers to trade and industrial teacher education 
programs ; 

2, for state certification, and 

3, for admission to an undergraduate or graduate 
degree program. 

Some states utilize these tests for one or two purposes; 
others for all three. 

The information received from the respondents, about the 
trade examinations they have prepared, is recorded graphically in 
Table III, The total number of different tests prepared foi . the same 
trade are indicated. This, however, is not the total picture. Tor 
example, a few states that have developed occupational competency 
tests and a- minister them, in most of the trades given in f.u Tabic, 
did not respond and, naturally, are not reported. The information 
does provide some keen insights into needs and the major trades in 
which test development is concentrated. 
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Table III - TRADES AND STATES IN WHICH OCCUPATIONAL COMPETENCY EXAMINATIONS ARE ADMINISTERED 



17a . 
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Research Project transferred from North Carolina to Illinois 

California has prepared many subtests in certain Occupations which are not listed here. 



III. 



DUPLICATION OF EFFORT IN TEST DEVELOPMENT 



Much of the work that has been done in the evaluation of 
occupational proficiency tests (outside the armed services) has been 
restricted by lack of adequate resources. Adequate funds have not 
been available to employ prof es i ional ly qualified personnel to work 
on the construction of the tests full time or (even more importantly) 
on the analysis of these instruments to insure the’* reliability and 
validity. The same applies to the preparation, duplication, adminis- 
tration of available tests, and the analysis of the results. 

A. Pioneering Efforts of Trade and Industrial 
Education Leaders 

The tremendous work in test development and administration 
of the former Bureau of Industrial Teacher Training and Vocational 
Curriculum Development of the New York State Education Department; the 
significant number of occupational competency tests and programs of 
the California State Education Department; the trade and industrial 
teacher testing program of the Pennsylvania Department of Public 
Education; and the pioneering work with subtests of the Division of 
Adult and Vocational Education of Florida State University, as well as 
the accomplishment of a few other states and universities, stands as a 
tribute to what a small dedicated group of men with minimal resources 
can accomplish. In spite of these singular contributions, every 
person, who has developed or administered such examinations, together 
with local, state, and national vocational indus trial- t echnical educa- 
tion, administrators, has expressed increasing concern for more re- 
liable measures to evaluate the occupational competency of prospective 
trade and industrial technical education teachers. 

Thus, this project reflects a significant breakthrough ... a 
long sought nationally coordinated effort to establish the current 
status, identify practical methodology acceptable to the states, 
develop and pilot test at least two major examinations on a national 
base, and th^n establish the feasibility of collectively pooling 
talent and resources through a cooperative consortium of states 
a r rang ement * 

While it was generally known that several states veue using 
occupational competency tests, the survey of existing practices has 
revealed that presently about the same number of states use tests 
as reported by Schaefer. In a number of instances, seveiaL centers 
within a state are engaged in test development. 

A simple suntrary of thirteen (13) occupational competency 
tests on which a number of states have concentrated their efforts, 
independently, with exceedingly limited (if any) coordination, is 
repotted in Table IV. As stated earlier, if all the states had reported, 
the researchers know that each of the numbers would be increased. 
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Table IV - Multiple Institution Activity Developing 
Occupational Competency Tests for the Same Trades 


Trade Area of Occupational 


Number of Institutions 


Competency Tests 


Preparing Similar Tests 


Machine " Shop 11 


20 


Automotive Mechanics 


15 


Electrical Construction 


12 


Electronics 


10 


Printing 


10 


Carpentry 


9 


Drafting 


9 


Air Conditioning 


8 


Welding 


8 


Cosmetology 


6 


Auto Body Repair 


5 


Diesel Mechanics 


5 


Sheet Metal Work 


5 



B > T he Use of Oral Testing Practices 

Most responses to the survey inquiry reported the use of 
written tests for evaluating trade proficiency. Some states also 
employed practical performance tests. Oral tests are still ustd by 
some states; others have discontinued them. Some of the oral tests 
were formal, containing structured patterns and predetermined questions 
intended for the candidate to supply information on his knowledge of 
trade information and trade practices. Other, more informal, oral 
interviews were designed primarily to assess a candidate's ability to 
express himself, his general bearing in a face-to-face rela t ionship, 
and his occupational and educational background. 

A small number of tests, included in Table IV, reflecL regional 
needs and requirements. However, most of the testr. are in trade areas 
which are little, if at all, affected by geographic or regional charac- 
teristics in skill, technique or special trade knowledge. This is to 
say, Ford, Chrysler or General Motors automobiles must be serviced the 
same way in Texas, New York or Maine. A machinist may work on different 
products in different states, yet he performs milling machine, lathe or 
grinder operations in the same manner, no matter where the equipment is 
located. Likewise, a radio set, a television set, amplification equip- 
ment or the various electronic devices found in radio or television 
stations require the same kind of service regardless of their geographic 
or regional location. 

Yet, the tabulation shows that for the machine trades, tests 
have been prepared in twenty different institutions located in nearly 
all sections of the country, /or the automobile repair and service 
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occupations, fifteen tests have been prepared; for electrical installa- 
tion, twelve; printing, ten; drafting, ten; and electronics, ten. 

Studies to validate the tests and establish their reliabil- 
ity have been undertaken by only three states. Lack of funds and 
staff with experience have restricted further efforts to carry this 
work to the level where the resulting instruments would provide objec- 
ive and reliable evidence for the evaluation of occupational compe- 
tence. 



The important question then is: "What results might have 
been achieved if all twenty institutions, who developed a machine trades 
competency examination, had supported a single effort of test construc- 
tion and pooled their knowledge and experiences to validate the results 
and assess hot; such instruments and the process of test development and 
administration might become more effective and efficient" . The same 
inquiry would relate to all other trade areas where a number of insti- 
tutions developed tests for the same occupations. 



Resources must be found to bring to bear on the whole pro- 
cess of occupational competence test development, particularly test 
development on the highest level possible of psychometric skill and 
theory, Such a concentration of practical test development is not 
likely to occur, as long as various states continue to work independ- 
ently on the problem, A coordinated research and development effort 
in which many states pool their resources would help to eliminate 
wasteful duplication, release important testing personnel for essential 
test development services and provide support for creative new 
approaches , 



Further support for such concentration of effort is provided 
by a content analysis of the available tests. The machine occupation 
and electronic test samples were analyzed as to the currency of test 
jontent, range of items, types and distribution of items, clarity of 
items, practicality and comprehens ivenes s and evaluated as poor, 
average, excellent. 



IV, ANALYSES OF AVAILABLE TESTS IN SELECTED 
TRA‘)E AREAS 



The written (theory) and perf ormance (industrial practices 
within the trade) intended to cover essentially the same body of 
content. However, even a casual check of the tests revealed a wide 
disparity of range in the selection and the number of items for a 
particular phase of the occupation. Thus, it became necessary to 
make a more careful analysis of the available tests in two major 
occupational areas. 

A. Machine Industries Occupatio ns (Machin e Trades) 

As indicated earlier, states using occupational competency 
tests may apply them in different combinations to various purposes, 
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Whichever combination of purposes are selected, the ultimate objective 
is to establish evidence of an individual's occupational competence. 

Since no descriptive information or statistical data has been made public 
by the various test developing agencies, a review committee of test and 
subject specialists was appointed for the purpose of assessing a number 
of written and performance tests in the same trade area. 

The major elements used as criteria in the assessment of each 
written test appear in the left half of Table V. Those that apply to a 
performance in simulating industrial conditions and practices are in- 
dicated in the right-hand portion. The information reported on these 
forms, as had been indicated in the transmittal letter, will be treated 
as confidential and no state, or a particular state examination, will bo 
identified. The comments made under the column "Value to Project 1 * were 
those of the project staff primarily. The intent was to avoid dupli- 
cation of effort by later using appropriate, effective and well prepared 
test items and techniques in developing the pilot tests. 



Table V - Summary of Evaluation of Machine Trades 
Written and Performance Occupational Competency Tests 



Written Test (A) Performance Test (B) 



Factors 

Considered 


*E 


A 


F 


Value 

to 

Proj ec t 


Factors 

Considered 




A 


P 


Va 1 u c 
to 

Pro i ec t 


Test Content 
(Currency) 


2 


4 


6 




Test Content 
(Currency) 


1 


2 


2 




Range of Items 


2 


5 


5 




Practicality 


- 


3 


3 




Types of Items 


1 


5 


6 




Comprehensiveness 


- 


3 


2 




Dis tr ibu t ion 
of Items 


3 


4 


3 




Skills 

Measurement 




2 


3 




Clarity of 
Items 


2 


3 


7 




Judgment 

Measurement 


_ 


2 


3 




Trade Judgment 


2 


2 


8 




Trade Diagnosis 


- 


2 


3 




Practicali * 


2 


2 


8 




Speed Measurement 


- 


3 


2 




Comprehens lve- 
ness 


1 


3 


8 




Procedures 

Planning 


_ 


3_ 


2 





Approx. % * V E“ Excel lence 

Objective A-Average 

Essay P-Poor 



B . Machines Industries Occupation 
(Machine Trades) - Written Test 

There was an exceedingly high degree of agreement amont the test 
evaluators as represented by the single composite number indicating the 
evaluation of a test for each evaluation item. No one test was judged to 
be excellent in all categories or all poor in all categories, The ratings, 
while representing a subjective judgement, are those of persons who arc- 
experienced in all aspects of occupational competency test development, 
administration, and evaluation. 



The distribution of the ratings indicates the need for more 
effective and reliable measuring devices and information to appraise 
a test. In che absence of statistical or empirical data, the committee 
determined the distribution of the test items among the various major 
divisions within the occupation and related trade areas. 

These major divisions within the machine trades and the 
related areas of technical knowledge dealing with blueprint reading, 
trade science and mathematics, etc., for which test items appeared in 
the tests that were evaluated, are reported in Table VI under these 
major divisions. The major divisions are referred to in some states 
as sub-areas. 

It should be noted that all but one of the tests used multi- 
ple choice type test items. Five tests contain only multiple choice 
test items. Four tests use multiple choice and true and false test 
items. One test utilized multiple choice, true-false, completion, 
and short answer types of test items. One test contained true- false 
and essay type items. The grouping of test items, according to the 
trade content areas they were to measure, revealed startling variations 

1. Only two of the wriiten tests included blueprint 
reading. 

2. Three tests attempted to measure competency in the 
theory of machine sawing and filing operations. 

3. Seven tests included items on industrial practices 
as experienced in the trade. 

4. Two tests include problems requiring the use of 
handbooks as experienced in the trade. 

5. Two tests contained problems and processes relating 
to numerical control or instrumentation applications 
in the trade. 

Within the structure of each of the tests, there was a 
tremendous disparagement in the range and number of test items. While 
all tests Included items relating to bench work, turning operations, 
drilling and drill press operations, grinding practices, c-nd milling 
machine processes, the wide variation in numbe of test items pur- 
ported to measure occupational competency, are summarized in TabLe VII. 



Table VII - Disparity in number of Test Items in 
flavor Divisions Covered in all Tests 


Major Divisions of the Trade 


Variations in 
Number of Test 
1 1 ems 


Bench and Assembly Work 


2 to 25 


Drilling Machines and Processes 


6 to 22 


Turning Machines and Processes 
(Lathe Work) 


4 to 78 


Grinding Machines and Processes 


3 to 34 


Milling Machines and Free esses 


4 to 26 
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The sura total of the multiple choice test items ranged from 
65 on the test with the lowest number to 253 on the test with the 
highest number. 

No effort was made to evaluate the quality of the test items. 
The test item tabulation merely shows a great lack of consistence in 
establishing which major divisions of the machine trades are most, 
typical and representative of the occupational competence o the journey 
men working in the machine industries. At first, it was thought. that 
the difference in content was the result of geographic characteristics, 
variation in products of the industry, and differences in training 
programs. A further inquiry proved this not to be true. 

The picture was made more confusing (even contradictory) wpen 
a review was made of a number of trade analyses. Made by machine trades 
teacher candidates in trades and industries teacher training programs 
in different regions of the ountry, and with widely differing expert- 
ences, the trade analyses showed a great consistency to the occupationa 
content which each individual reported as representative of the compe- 
tencies a journeyman in the machine industries should possess. 

C . Mach ine Industries Occupations (Machi ne Trades)_ 

Per f ornance Tests 

The criteria for evaluating the performance tests were 
earlier identified as appearing in Table V. As implied, the perform- 
ance test.- concentrate on the day-to-day work skills and directly 
applied technolrgy which a journeyman tradesman performs. The tests, 
therefore, siould measure such characteristics as skill, speed, accu- 
racy safe work habits trade judgement, diagnosis capability and those 
other factors which identify a competent journeyman tradesman. 

Only five tests were readily available for analysis. Note 
that only one of the performance tests was judged to be excellent in 
only one factor; namely, currency of content. 

An analysis of the trade operations essential to complete 
the test work piece(s), as well as a study of the machines tools and 
equipment on which trade skills were to be measured showed a heavy 
concentration on turning processes and lathe work. One test involved 
producing several short work pieces: one, requiring operations, n 
ling and other bench work, drill press processes, surface grinding 
and simple milling machine work and operations. «notner consisted of 
turning, milling and universal grinder operations. The other P e ^r- 
mance tests concentrated on turning processes and engine lathe work., 
simple milling operations, and elementary layout work- -usual ly 
requiring the making of one more work piecc(s). 

There is no evidence available which would indicate that 
occupational competence in the machine trades can be measured hy 
acceptable performance on turning and milling processes and machines, 
bench work and assembly operations. Quite the reverse is indicated 
by trade analyses, job specifications and tne demands of journeymen 
craftsmen in the machine trades. In addition, the machine trades 
teacher must be capable of teaching industrial processes for the full 
range of machine tools, heat trealment, and bench and inspection 
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practices which are common to the trade. The level of teaching ranges , 
from classes of beginners, preparing to enter the machine trades as 
advanced learners/apprentices or technicians, to special groups of 
journeymen craftsmen who return periodically to the schools for occupa- 
tional extension work to upgrade their skills and technical competencies. 

This analysis of written and performance tests for the machine 
trades indicates severe limitations in test development, and the need 
for more concentrated, coordinated effort, a broader base for the test 
items, and greater specialized test development competence. These are 
necessary to develop tests that are relevant to current industrial 
practices, have acceptable accuracy, validity and reliability, and can 
be utilized by a consortium of states interested in a cooperative 
endeavor . 



D. Electronic Industries Occupations 
(Electronics - Communicat ion) Tests 

The same factors were applied by another selected committee 
for electronic communications. The committee assessed seven (7) written 
tests that were submitted and four (4) practical tests. The results of 
this assessment are summarized in Table VIII. 



Table VIII - Summary of Evaluation of Electronic Communications 
Written and Performance Occupational Competency Tests 



Written Test (A) 


Performance Test (B) 


Factors 

Considered 


*E 


A 


P 


Value 

to 

Proj ect 


Factors 

Considered 




A 


P 


Value 

to 

project 


Test Content 
(currency) 


1 


4 


2 


1 


Test Content 
(currency) 


1 


2 


1 




Range of Items 


3 


3 


1 




Practicality 


1 


2 


1 




Types of Items 


2 


5 






romprehensiveuess 


- 


3 


1 




Distr lout ion 
of Items 


1 


5 


1 




Skills 

Measure 1 ent 


* 


3 


1 




Clarity of 
I terns 


2 


4 


1 




Judgment 

Measurement 


1 


3 






Trade Judgment 


1 


3 


3 




Trade Diagnosis 


1 


3 






Practical icy 


2 


3 


2 




Speed Measurement 




3 


1 




Comprehensive- 

ness 


2 


3 


2 




Procedures 

Planning 




3 


1 





Approx. 7„ >v E - Excellence 

Objective A Average 

Essay P ' Poor 
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Totals of Te.s _ Test A Test U Test C Test 

I Cents by Tests 275 11C 91 US 



Again, no one tee*: was judged to bn either excellent or poor, 
as a whole , Some categories received j grading uf excellent; Other 
sections were judged average or poor. ihese ratings, again, indicate a 
substantial degree of inconsistency in tost development and administra- 
tion, and the need for joint national effort to inspire all aspects of 
occupational competency testing. 

E, Electronics Industries Occupations 
(Electronics-Communic at ions) Written Tests 

On the written test, ail test items were phrased in a combi- 
nation of multiple choice patterns. A casual glance at Table IX, which 
tabulates the subdivisions from which test items were derived, shows 
substantial difference in concept and judgement as to what constitutes 
occupational competence. The content of one test bore close resemblance 
to textbook outlines, whereas two of the others were developed around 
equipment commonly used in the trade and in studios, The number of test 
items on these tests ranged from ninety-one (91) to two hundred and 
seventy-five (275). No evidence is available which supports that one 
test is better than another. Also, there is no information based on 
test experience as to how many, and what kinds of test items, constitute 
a valid, tellable tost. 

All tests contained a section on "Basic 1 information. The test 
items listed under fundamental concepts, basic electricity, fundamental 
principles and electronic theory, dealt with information most often found 
in high school or two-year institute textbooks on physics and science. 
There were few test items relating principles to applications performed 
by the craftsman/ technic ian in the occupational world. 

For tests intended to measure occupational competence in an 
occupation in which geographic or regional differences plan an insigni 
ficant part, the variation of categories in the tests is amazing and 
seems to reflect the biases and preferences of the item writers to use 
a textbook, theoretical approach, even to the point of dealing with 
academic sciences, rather than the results of a thorough analysis of a 
cluster of jobs in the electronics communicat ion field. A comparison 
of the test content with trade analyses made by teacher candidates in 
trade anc industrial/technical teacher education programs, with diverse 
experience backgrounds, showed much stronger consistency z i to what 
constitutes journeyman competence among the analyses than among the 
tests . 

F. Electronics Inductries Occupations 

Performance Tests 

The performance tests concentrate on "troubleshooting" under 
simulated conditions. Defects and faults are introduced intu pertain 
nieces of equipment which the examinee must correct. This approach is 
representative of the w T crk in the field. Referring back to Table VIII, 
note that no test was judged excellent or poor in Its entirely. There 
was a greater co;:non core, or uniformity of subtrade est areas, among 
the electronic communications performance tests than for the machine 
trade tests. The evaluation of the electronic communications practical 
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tests concentrated upon procedures and methods rather than entirely upon 
the finished product. No supportive data for test evaluation was avail- 
able to the project. 

The end results or products produced through the testj although 
limited in number, give strong support to the need for much more concen- 
trated effort in test development. All evidence points to needs within 
state departments tJ education, and trade and industrial teacher educa- 
tion institutions for sufficient funds and adequate professional and 
support staff, facilities, and equipment to carry on all facets of occu- 
pational competency testing. The planning, test development, adminis- 
tration, evaluation and research necessary for establishing of validity 
and reliability across a wide base of national testing can not be done 
under existing conditions. The reporting, and tabulating of results 
necessary to establish valid national standards, which the states may 
apply to their special needs, require broad support. 

V. OBSERVATION RATINGS, ORAL TESTING 
AND TESTING TIME 

A i Obs ervation Ratings of Wo rk Skills and Ha b it_s 
in Per forming Examinations 

For the performance tests, the emphasis was Primarily on real 
live situations with the finished product the measure of a candidate 1 s 
occupational competency. However, it was recognized that the work skills 
and processes in achieving a finished product are important factors in 
the overall proficiency appraisal. A number of states, therefore, in- 
cluded factors to be observed while the candidate was at work. 

This personal observation by a trades examiner introduces a 
number of variables in the performance evaluation. While it is rela- 
tively easier to establish criteria for written tests, the observation 
of work habits, work attitudes, workmanship, safe practices, etc. involve 
subjective judgements which affect the reliability cf performance tests. 
In two tests, several specific items were listed with a range of point 
values assigned to each judgement. The finished product evaluation 
involved accuracy and completion of a number of steps within time limits. 
A graduated range of credits was assigned to record superior accuracy 
and completion in shorter time when the candidate performed less satis- 
factorily in terms of quality, accuracy and time requirements. 

B. Time Allotment for Condu c ting Examinations 

With very few exceptions, the written tests were objective in 
structure. Multiple choice and true- false items constituted the major 
part of written examinations. In some instances, completion and short- 
answer essay- type test items were included. The reported time span for 
written examinations ranged from two to four hours. For the performance 
tests, a longer period, ranging from four to eight hours, was needed. 

No time limit was indicated in those instances where the oral test, or 
interview, was given. Kazanas and Kief have reported time spans for oral 
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and interview tests from one-half hour to tvo hours. Based on observa- 
tion and comments by individuals who relate to the interviewing and 
selecting of trade teachers, a general estimate of one hour for the oral 
part provides sufficient time to establish a candidate's background and 
to ma^ e other subjective judgements. 



VI. RESTATEMENT OF PURPOSES OF OCCUPATIONAL 
COMPETENCY TESTS 

Earlier, it was stated that occupational competency tests are 
used to: establish a teacher candidate's degree of occupational profi- 
ciency; meet certification requirements, fulfill one of the requirements 
for entrance into trade and indus tr ial/ t echnical teacher education pro- 
grams and instill others to serve as a determining factor for earning 
college credit in undergraduate or graduate degree programs. 

Sumn: irizing what local and state adrruuis t rator s and trades and 
ind'^ti es teacher educators have reported, occupational competency test 
results are utilized for part, or all, of these purposes: 

1. As a regular means for certification, 

2. As a requirement for probationary appointment, 

3. As a prerequisite for admission to preei. A loyr n t 

industrial teacher education prog re , 

4. As a basis for granting college credit in a content 
(specialization) area, 

5. For prognostic and diagnostic purposes to establish 
weak or deficient content areas which the candidate 
needs to correct or improve, and 

6. As fulfilling one of several requirements for 
certification . 



VII. TEST WRITERS AND COMMITTEES INVOLVED 

IN OCCUPATIONAL COMPETENCY TEST PREPARATION 

Practically all of the occupational competency tests are the 
result of committee effort. The composition and the backgrounds of groups 
developing occupational competency tests, as reported, include the follow- 
ing combinations: 

1. Competent teachers in the specific trade or industrial 
occupat ion ; 

2. Selected occupational specialists from industry 
working with teachers; 
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3. State trade and industries education supervisory 
staff; 

4. Trade and industries teachers and state officials; 

5. State depai aneni: test development personnel, trade 
and indus trial/ technical teacher educators, and/or 
state trades ard industries supervisors; 

6. Industrial teacher educators and occupational 
specialists from industry; 

7. Qualified teachers and specialists from industry, 
supervised by state department and industrial teacher 
educational personnel with the assistance of joint 
advisory committee members; 

8. Teacher educators, qualified vocational teachers and 
specialists from industry. 

Variations in the composition of the groups involved in 
developing the tests naturally effect the nature and administration of 
the evaluative instruments and influence their quality. 

vrrr. state practices in test administration and eva dation 

A. Administration of Occupational Competency Tests 

Reports submitted by the states and teacher education insti- 
tutions reveal that competency examinations are administered by: 

1. Trades and industries teacher education personnel 
working in institutions. 

2. Trades and industries state supervisors from -.tate 
education departments, 

3. Testing supervisor (reported by onp state). 

Three industrial cities have assigned the responsibility of 
evaluating the competency of trade and indus tr ia 1/technica 1 teachers to 
a special board of examiners. 

B, Evaluation of Test Results 

Another serious problem in establishing uniform, valid 
practices for measuring occupational competency is represented by the 
manner of grading the tests and the individuals who carry out this 
responsibility. 
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The objective types or parts of test are graded by: 

1 „ Secretary in the. office Oi supervisor or trades 
and industries, 

2, The same persons who constructed the test, 

3, A teacher who administered the examination > 

4, State trade and industries supervisory staff, 

5, .Trades and industries teacher educators, 

6, Qualified proctors specially engaged to administer 
ana grade the examinations. 

Several states reported using data processing for the grading 
process and for reporting written test results. Other institutions are 
presently engaged in trying to establish data processing procedures for 
reporting the results of performance tests. 



IX. DATA PROCESSING CONSIDERATIONS 

It should be noted in this connection that the project 
staff has been charged by the Planning Coirmittee and Principal Investi 
gators to utilize data processing. Thus, consideration is being given 
to techniques in test development, measurement analysis which may be 
computerized into a test bank storage, retrieval, and distribution 
system. 



When functional, such a system will be applicable and avail- 
able to every participating organization in the Consortium of States, and 
ensures accuracy of results, more efficient use of test development an I 
measurement personnel, and greater efficiency in making print-outs of 
results for diagnostic and other purposes. 



X. VALIDITY AND RELIABILITY OF TEE TESTS 

Tne validity and reliability facets of the study, unfor- 
tunately, must be reported with some brevity. The evidence is crystal 
clear. The surface of need for establishing validity and reliability 
for occupational competency tests In the trades and industries has been 
barely scratched, This fact will later be borne out and reemphasized as 
a condition existing in testing programs of other groups representing 
military establishments, industries and labor organizations. The exper- 
iences of private educational testing and psychological testing services 
is identical. 



Two states reported ground-breaking work in establishing 
validity and reliability of occupational competency tests through studies 
utilizing available data. No other information has been found in the 
literature to support further effjrts in this area. However, a number 
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of tests have been subjected to *uch statistical treatment, as: item 
analysis, difficulty of items, etc. In general, face and internal 
validity of tests was based on "the authority of the expert". The 
authorities were experienced tradesmen, teachers, teacher educators, 
state supervisors all of whom had been involved in analyses and develop- 
ment. Other bases were found in appropriate texts and other technical 
publicat ions . 



The obvious fact is that occupational competency examina- 
tions, in the trades and industries, have not generally been constructed 
and the results validated or standardized according tc acceptable pro- 
cedures of test evaluation. 



XI. CONSTRAINTS TO A COORDINATED OCCUPATIONAL 
COMPETENCY TESTING £ f FoR T 

It should be reported that only adaptations of known pro- 
cedures and techniques for test planning, development measurement, 
evaluation or reporting have been uncovered in this survey. Analyses 
of the tests used by the military, industry, labor, testing services 
and others, reveal practices that have been used over the years by occu- 
pational competency test development persons in trade and industrial/ 
technical education. Trades and industries educators have been recog- 
nized for their extensive knowledge and practical expertise in raking 
trade and occupational analyses, utilizing most forms and kinds of tost 
items in constructing theory and performance tests, and have the capa- 
bility and working knowledge to adapt tests to computerized procedures. 
All of what has been uncovered has been utilized by the trades and 
industries educator in one form or another. 

The central issues focus on seven (7) major forces that 
constrain these professional educators from making a broad base and 
an in-depth approach to the problem. Although, there arc other con 
straints, the survey identifies seven (7) restrictions which vitally 
affect the process and product of occupational competency testing for 
the trades and industries: 

1. Lack of permanent positions in test development and 
measurement, with little encouragement towards a 
full-time career in this program, 

2. Limitations of financial resources and full-time 
personnel to select, coordinate and organize 
committees. 

3. Scarcity of qualified researchers and testing 
specialists with adequate time to actually develop 
good examinations and follow through on their analyses 
to establish reliability and validity. 

4. Tack of vehicle to cooperatively administer the tests, 
and conduct other essential eval«* rtiou function*, 
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5. Lack of a central physical facility aud limited 
personnel for conducting continuous validation and 
reliability procedures required to truly make tests 
objective instruments for competence evaluation, 

6. Restricted geographic bases and essential national 
test population to establish appropriate norms, 

7. Limited coordination of test development even in 
states where two or three organizations are engaged 
in such work. 

The early efforts of test construction and administra t ion, 
which had been made under discouraging circumstances and conditions, 
still prevail today when this nation is at its highest point in indus- 
trial development and technology. This in-depth review of the state- 
of-the-art has shown much duplication of effort and concentration on 
test development for a limited number of trades. It has, also, re- 
vealed a lack of supportive statistical data upon which the accuracy, 
validity and reliability of a test may be established. 

It is, therefore, not surprising that a number of dedicated 
individuals, in a number of states, who are concerned with the pro- 
fessional improvement of teachers in vocational education are urging a 
national coordination of occupational competency testing effort. The 
program should be of sufficient scope and flexibility to meet the 
special needs of all the states on a voluntary basis, and should produce 
valid and reliable instruments through which it is p >ssi lo to establish 
standards and norms of occupational competence, and to qualitatively 
affect the total services provided through vocational education. 
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OCCUPATIONAL TESTING MATERIALS AND PROGRAMS 01 -INDUSTRY, 
1A30R AND GOVERNMENTAL ORGANIZATIONS 



I . 



SURVEY OF OCCUPATIONAL COMPETENCY TESTING PROGRAMS 



OF INDUSTRY 



Sources of Industrial Contacts 



Future teachers of traue, industrial and technical subjects 
acquire ana develop their skills in industry. The placement of individ- 
uals with skills and know-how appropriate and adequate to meet the re- 
quirements of a particular job in industry has become an economic neces- 
sity. It seemed, therefore, highly desirable for the project staff to 
investigate the methods and procedures employed by industrial firms for 
the evaluation of occupational competency to see whether similar pro- 
cedures would be applicable to the evaluation of the same competency 
essential for teaching in vocational programs. 

The research team was faced with a difficult task. From among 
the many thousands of industrial establishments, it vrs necessary to 
determine which companies do engage m formal performance testing and 
the industries in which they are located. 



The review of the published lit 
studies whatever on performance testing, 
the American Management Association, The 
Manufacturers, the Industrial Conference 
of Industrial Management, National Metal 
a ted Electrical Industries, National Tool 
Association and the National Association 



erature revealed no printed 
Assistance was sought from 
National Association of 
Board, American Association 
Trades Association, Associ- 
, Die and Precision Machining 
of Industrial Training Directors 



The National Association of Manufacturers opened their files to 
the research staff and a list of companies was compiled from various 
industries, who conduct some form of personnel evaluation through tests. 
It was, also, possible to obtain the names of individuals engaged in 
employee testing and personnel research. A mailed inquiry was sent to 
thirty-five (35) companies soliciting similar information as had been 
requested from each of the states. 

These companies were contacted by mail. The response was 
rather disappointing; of the thirty-five (35), there were twenty (20) 
respondents; of the twenty (20) companies, two (2) carried on a program 
of occupational competency testing. The remaining eighteen (18) replies 
were negative, indicating no formal competence tests were administered. 
The responses indicated that occupational competency evaluation for 
upgrading, or for salary or wage classification, is largely the responsi- 
bility of the immediate supervisor whose judgement, based on the day-to- 
day job performance observation, is accepted as valid evidence of an 
individual's competence. Of special interest to the project were the 
two companies which do carry on occupational proficiency testing. One 
company administers a written proficiency examination. The examination 
content is derived from an analysis of the various job descriptions 
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which are part of the employe e- management contract. No efforts have 
been made to carry on statistical treatment of the test results. The 
other company is carrying cn simulated performance tests based on a 
thorough analysis of the tasks ^n a particular job. Other forms of 
tests are presently in the development stage. The American Management 
Association recommends a persona) appraisal form to its members which 
includes personality characteristics as well as a rating scale of job 
performance . 



B. Uses of Tests by Industry 

A review of several studies published by the National Indus- 
trial Conference Board show that tests are used in industry for 
determin ing: 



Mental ability (Intelligence) 

Mechanical ability or aptitude 

Interest inventories of employees relate to measures 
of personality 

No evidence was presented that any of the three hundred 
eighty-four (384) firms responding to the National Industrial Conference 
Board’ s questionnaire relating to occupational competency testing carried 
on any f/'im of practical performance testing. The tests the companies 
were using applied to white and blue collar workers. Interest and 
personality evaluations were used to a much larger degree with white 
collar workers; mechanical- aptitude and mechanical ability ^ests, with 
blue collar workers. 

Stated briefly, tests are being used for predictive purposes 
rather than for the evaluation of an individual's job proficiency. It 
is, therefore, not surprising that most test applications in industry 
are for selection and placement at the time of beginning employment. 
Approximately, 10"% of the firms surveyed utilized tests for counseling. 
Among the nearly four hundred (400) companies using one or more tesLs, 
sixty-five percent (65%) use commercially available tests. Of the 
companies with over one thousand (1000) employees, seventy-nine percent 
(797o) use these conmercial tests. Those companies with less than one 
thousand (1000) employees, fifty-three percent (53%)- In addition, 
nineteen percent (197.) of the total companies included in the NICB 
survey apply commercial ly availaole tests for white collar employees 
only; five percent (5%) for blue collar employees only. 

Twenty-six percent (26%) of the companies, employing one or 
more tests, use their own tests (or those developed through cooperation 
with a trade association) for both blue collar and white collar em- 
ployees. Such tests are used by seven percent (7%) of the companies 
for white collar employees only; seven percent (7%) for blue collar 
employees only. 
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C , Personnel Respons ib le for Industrial Tes ting 
Materials and Prog rams 

Of interest to the project staff were the background and 
training of the individuals in charge of industrial testing programs. 

A breakdown of the job classifications of such persons or agencies for 
two hundred thirty (230) manufacturing and non-manufac tur ing industries 
is reported in Table X. 





Table X - Job Classificati 
for lndustria 1 


ons of Personnel Responsible j 

Testing Programs 






Mature of 


industry 




Job Classification or 
Testing Agency 


Manu factor ing 


Xon-Mcnuf ac tur ing 


1 


Company employee with 
training in psychology 


59 


75 


2 


Company employee who is a 
qualified psychologist 


9 


6 


3 


Director or supervisor of 
personnel 


5 


16 


4 


Mot specified 


12 


24 




Outside testing service 


15 


9 




Totals 


100 


130 



D > Observations on Limited Industrial Proficienc y 

Testing 

There are many reasons why there is so little work published 
or, in fact, actually done in occupational competency testing in indus- 
try. Possibly, industry believes it satisfactory and has found it 
expedient to follow the practice of closely observing the performance 
of new employees during their beginning days of employment as the equ _ v- 
alent to a performance test. In the absence of standardized, valid 
instruments, which may be secured commercially, many medium and small- 
sized companies have stated that lack of qualified competent personnel, 
and the cost of developing accurate and reliable instruments, makes the 
operation of a performance testing program prohibitive. Whether action 
to not carry on an occupational training program is justified on such 
grounds is a decision industry must make. 

However, the project staff who have spent their industrial 
and professional liv r cs working in and with industries, ir. expressing 
their personal beliefs, feel that valid occupational competency programs 
should be widely established across all industries, in the labor movement 
and throughout government agencies. 
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The Investment by industry in such programs would pay great 
dividends in human satisfaction and in increased productivity. Certainly, 
the high cost of limited productivity of the untrained or untrainable; 
the man hours lost through underemployment; the failure to recognize 
occupational skills and technology developed within quality programs of 
vocational education by employing tested graduates as advanced learners, 
ate just a few of the reasons why industry should assess and expand its 
efforts in occupational competency testing. The most critical occupa- 
tional sector of the economy relates to the blue collar craftsmen and 
technicians in the skilled trades and industrial/ technical occupations. 



II. SURVEY OF OCCUPATIONAL COMPETENCY PROGRAMS OF CIVIL SERVICE 
AND OTHER GOVERNMENTAL AGENCIES 

A. Changing Concepts of Comp e tency Testin g 

Government agencies, as well as any other organizations em- 
ploying people for jobs which require a wide range of skills and con- 
siderable training, are vitally concerned with the selection of the 
individual with the appropriate occupational competency for the right 
job. Civil Service agencies, on national, state and local levels, 
charged with the selection of personnel for the multitude of jobs found 
at all employment levels, have employed examinations in various ways 
over a period of many years. Most Civil Service examinations are, how- 
ever, of the paper and pencil variety ano cover primarily the job 
knowledge, job information and job judgements essential to carry on the 
work of a particular job classification. 

For many years, it was believed that there was a high degree 
of correlation between occupational information and job performance 
involving manipulative skills that a command of the theory could serve 
as an indirect measure of skill competency. Many experiences gained 
during World War II, with assessing shop skill performance, disprove 
this theory. The research conducted by Stuitt, in appropriately 
weighing the relationship between job performance and tested job 
knowledge, in the final competency evaluation proved that written 
tests of occupational knowledge are not dependable or good measures for 
the evaluation of shop skills. An accurate assessment of an individual's 
occupational competency can not be made without some type of direct or 
indirect performance measure (Shimb erg) 27 . These same conclusions were 
reaffirmed last year in an earlier reported study of Koenigsberg and 
Reilly. 



In recent years, the evaluation of individuals for jobs, 
requiring a degree of skill in office occupations, have included per- 
formance tests. These office and business occupations tests have been 
standardized and are frequently used to measure several levels or job 
clusters of job classifications. 

In those trades and industries occupational areas where manip- 
ulative skills arc the basic requirement for job, performance tests arc 
part of the overall evaluation of an applicant's qualifications. Auto- 
motive maintenance, building ma intenance , machine occupations and 
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electrical occupations are typical of those areas requiring performance 
tests. 



fi . Construction of Civil Service Te sts 

A st^ff of test specialists determines the scope and content 
of the examinat ions from a job analysis, based on the specifications 
furnished by supervisors of the departments in which positions must be 
filled and from other experts in the field. Emphasis in the perform- 
ance examination is placed on the practical requirements of the job. 

Some statistical effort is made to establish validity and reliability 
through item analysis, pretesting of questions, measures of central 
tendencies, means, standardization, and others. However, limitations 
of personnel, the great numbers of examinations which have to be pre- 
pared place heavy reliance upon face validity and internal validity, 
and support by the authority of the expert. With the availability of 
computer facilities, it is hoped that more effective efforts can be 
made in standardizing performance tests, in establishing norms and 
pursuing validity and reliability studies. 

Passing grades are emperically established and are influ- 
enced by the supply and demands of the labor market. Attempts have 
been made to establish a bank of test items. Here, again, limitations 
of personnel and funds make it difficult to keep the test item pool up 
to date. For those occupational areas where the permanent testing 
staff lacks the expertise to develop examination items, outside experts 
are engaged to provide the necessary examination content, 

C , Administration of Performance Teats 

The problems encountered in the administration of perform- 
ance tests by Civil Service Departments are similar to those of all 
other organizat ions engaged in occupational testing. For instar.ee, 
in New York State, the examinations are given in selected centers which 
lend themselves to the conduct of the tests in a realistic and practical 
set ting--area garages and repair centers, shops, and laboratories 
typical of the work situations. Occasionally, the shops of vocational 
education centers or technical institutes are used. Frequently, the 
supervisors of the departments act as examiners. Specific directions 
for evaluating the applicant's performance are issued and examination 
results are reported by each examiner independently in writing. 

Special care is exercised to maintain examination security 
for those tests which must be given in different locations at different 
times. Selecting and instructing examiners is a common problem, 

D. Observations on the Adeq uacy of Civil Service 
and Other Government Testin g Programs 

Contact with various bureaus of the Civil Service at the 
federal, state and local level involved in test development and adminis- 
tration reveals a similar approach to the problem of occupational compe- 
tency testing used by industry. Much of the testing is still in the form 
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of written tests. However, there is increasing recognition that jobs 
involving a high level of skill require performance tests to extablish 
an applicant’s occupational competency. 

Efforts are being made to develop such tests on a more 
objective basis. Limitations of personnel and funds and the tremendous 
range and number of positions for which examinations must be prepared 
have slowed the development of performance tests. Lack of proper facil- 
ities, the limited number of applicants for a single trade examination, 
the need for conducting these examinations in various geographic loca- 
tions within a state or throughout the nation, places added restrictions 
upon performance test development and administration. 

In larger metropolitan areas, where there are heavy demands 
in certain fields, test centers have been established and are functioning 
within certain limits. The concensus of the individuals contacted, re- 
garding the value of presently administered performance test, ranges 
from acceptance of existing practices the tests do the job for us... 

they are practical and we have few challenges..., 1 ') to recognition that 
a great deal of research is necessary to develop accurate, valid and 
reliable instruments. Surprisingly, little information concerning 
performance testing has been published for the vast number of Civil 
Service testing programs, 



III. SURVEY OF OCCUPATIONAL TESTING PROGRAMS OF MANPOWER DEVELOPMENT 

AGENCIES 

The search for occupational competency testing included 
agencies and organizations involved in manpower training and develop- 
ment, various state and local employment agencies and private institu- 
tions. No information was available on any form of occupational per- 
formance testing, State and 3 /cal employment services, at times, 
employ tests concerned with mechanical aptitude and comprehension, 
manual dexterity, spatial visualization, mechanical and abstract 
reasoning, intelligence and other factors. These are utilized fo** 
predictive purposes. 

Some tests involve peg boards, tweezers, assembly of 
simple parts or blocks, and the like. Most tests are of the paper and 
pencil type. Perhaps, in the manpower field with its emphasis on 
training and development, there is greater urgency for determining 
potential tather than to establish instruments for evaluating occupa- 
tional proficiency. 

There is little doubt, however, that the increasing 
complexity in the skilled occupations and the risk involved in placing 
less than competent workmen in these occupations will eventually re- 
quire some form of performance evaluation. Prediction of potential 
success is significant; evaluation of competence is equally, if not 
more important. 
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IV. SURVEY OF OCCUPATIONAL COMPETENCY PROGRAMS OF ORGANIZED LABOR 



A. Wo rk ing Relationship Between Or ganized Lab or 
and Voca tional Indust r ial Educa txon 

Vocational education and organized labor have, over the 
years, maintained a close relationship. Union representatives are 
always included on state and local overall vocational advisory boards, 
and actively provide advice and counsel as members of craft advisory 
committees. They are, also, represented on state and local joint 
apprenticeship committees and on national professional committees, 
such as the Trade and Industrial Education Divisions Lab or -Management 
Advisory Council for the American Vocational Association. 

Many industrial and craft unions carry on intensive edu- 
cation and training programs through special training departments within 
their organizations. In many states, a close working relationship is 
maintained between the state apprentice training . supervisor and the 
various unions for which formal apprentice training programs are 
conducted. The United States Government, through the Department of 
Labor and its interdepartmental relation.^ ip with the United States 
Office of Education, Division of Adult anu Vocational Education, is 
concerned with apprentice training, as well as the conditions of em- 
ployment and levels of competency in occupations. 

B . Limited Occupy ional Competency Testing Progra ms 

Contact was made with representatives of national apprentice 
ship committees and requests submitted regarding the procedures and 
methods employed in evaluating the competency of apprentices when they 
reach the level of journeyman competence. A number of locals were also 
contacted with requests for the same information. 

The replies received proved to be disappointing. Evalu- 
ations are made on a judgement basis on the iob. The research team 
was unsuccessful in locating any formal testing program, or any tests 
prepared in addition to those constructed by instructors in training 
programs. It was concluded that there is a scarcity, if not a void, 
in the development or administration of special formal occupational 
competency tests. 



V. SURVEY OF OCCUPATIONAL COMPETENCY TESTING 
P ROC R AI'IS OF THE MILITARY 

A. General Observations 

Under the Department of Defense, the United States Army, 
Navy, Air Force and the Coast Guard carry on one of the most diversi- 
fied and complex training programs ever devised. In a multitude of 
Bases and Training Centers in the United States and around the world, 
training programs range from the development of simple office skills 
to the operation, maintenance and repair of extremely complicated and 
sophisticated technical equipment. 
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The very practical aim of all these programs is to develop the 
individual soldier, sailor or airman to perform his assigned tasks with 
competence and the highest possible degree of proficiency. Thus, evalu- 
ating proficiency is a matter of serious concern for those charged with 
the training and preparation of personnel. In fact, testing and evalua- 
tion personnel in the armed forces has grown to the point where, for the 
last eleven years, a Military Testing Association has served effectively 
to conduct annual national conferences in which both military personnel 
and civilian professional personnel participate. 

B_. Predominance of Paper and Penci l Types of Competency Tests 

In reviewing the literature, ic was found that relatively few 
papers, regarding performance tests, have been published. Perhaps, this 
is due to the highly specialized nature of the research carried on. The 
survey team, attempting to obtain information on existing practices of 
competency testing, made on-site visits to a number of military training 
centers in which tests were developed and administered. 

It was soon recognized that, although a great variety of tests 
are used, most of those adminis ter ed are of the penciL and paper type, 
and that visitations would have to be confined to a selected number of 
Centers in which performance tests were employed. Among the services 
for the Army, the Signal Corps Schools, and the Mavy, the Electronics 
Schools were selected for visitation. 

Responses from the Air Force indicated that levels of competence 
of individuals were determined through specialty knowledge tests. Current 
Air Force Research concentrates on supervisory evaluation of job perform- 
ance and criteria development. 

Much of the testing in all branches of the military relates to: 
measuring the achievement of trainees in the various service schools, 
evaluating the validity of the instructional content of each curriculum, 
appraising the quality of instruction, and evaluating the background, 
ability and prerating preparation of the trainees. 

The discussions held with civilian advisors and military person- 
nel involved in training and competency evaluation, revealed a strong 
interest in performance testing. Unanimously, they voiced a need for 
extension and improvement of performance tests presently in use. Even 
those who were actually involved in their development expressed the need 
for more accurate and reliable instruments. 

C, Patterns for the Construction of Tests 
f or Military Services 

The service schools visited were organized with special personnel 
assigned to bureaus or sections for the development, administration, and 
control of written and performance tests. Test items are developed by 
the instructional staff, based on the content of a particular course. 

The test items are then reviewed by the test specialists for relevance 
and comprehensiveness, and the tests are used experimentally. Tests are 
then administered with trainees. 
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After revision, the test is constructed anc’ applied. Test 
results are computed and the data analyzed. Item analyses are nadc, 
scores are standardized, and mean scores, maximum and minimum scores, 
and passing levels, are established. Efforts are made to establish 
item pools for alternate forms. 

Where computer service and electronic data processing equipment 
is available, test item and test analyses are extended through more 
intensive validity and reliability studies, treatment of variables, and 
the development of norms. 

Performance tests are developed from task analyses derived from 
job descriptions. In one installation, a special section, fully equipped 
with apparatus and instruments, was set up for the development of per- 
formance tests. The objectives of a test are established for the purpose 
it is to serve. Then, a step-by-step analysis is conducted by actually 
carrying out each task. The items to be rated are determined and values 
are assigned to each item, according to its importance (skill, judgement 
and procedure) in completing the job. The testis constructed and direc- 
tions are developed for the trainee and the test proctor. 

Rating forms are prepared and the range for the judgement of the 
rater is indicated. The test results are then tabulated and recorded. 
Because of the unavailability of computer service, at this service center, 
much of the data is hand tabulated. This has placed limitations on the 
establishment of norms and the making of validity and reliability studies 
and treating the data statistically. While there is still a good deal of 
subjective evaluation in the final appraisal, by providing a fairly 
specific range for rating, the rating results tend to be more objective 
then was the case several years ago. Actually, the military testing 
program functions in a setting very much different from civilian life. 
Motivation and control are different. Nevertheless, the manner in which 
performance tests are developed, the rating scales applied, and the 
values assigned, provide a greater measure of objectivity than heretofore. 

D , Enlisted Evaluation System 

To attract men and extend their military careers, the Army 
has developed an examination system for the evaluation of comparative 
competency of enlisted active duty personnel* Established in 1958, the 
Enlisted Evaluation System has increasingly influenced promotion, 
training, assignments and increases in pay. 

Evaluation is accomplished by the use of a combination of instru- 
ments. A paper and pencil test is used to determine the soldier's ability 
to apply his knowledge, training and experience to the requirements of his 
military occupation specialty. The enlisted person's efficiency report is 
the supervisor's appraisal of certain personal characteristics, describing 
how well the individual actually performs on the job. Performance tests 
are used when appropriate. The written tests include performance type 
test items wherever possible, in order to determine how competent an 
individual is in his r'litary occupational specialty. 
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The evaluation system is supported by a worldwide network of 
over three hundred fifty (350) test control officers. There are thirty- 
two (32) item-wr it ing and ten (10) supervisory agencies that prepare 
test materials in the Test Development Process. The item-writing agencies 
are located within the United States Continental Army Command Schools, 
Department of the Army, and Department of Defense Organizations which are 
designated as responsible agencies for selected military occupations. 
Subject matter specialties from these agencies furnish basic reference 
lists for test aids and provide items or questions for the tests. 

The tests are administered in the field by a test officer. 

The test results are forwarded to the United States Enlisted Evaluation 
Center. An evaluation score is developed by combining the values of 
test results, the enlisted person's efficiency report, and performance 
test score, when used. An evaluation score compares each individual's 
relative standing with all others who have the same skill level and pay 
grade in the same military occupational specialty. The ultimate goal is 
to prepare a test for every military occupational specialty. 

E. Overall Observations of Military Testing F ro gram s 

A great deal of effort, time and money has been devoted to this 
evaluation of military personnel program. Test data have been statis- 
tically treated and, for a certain number of tests, validation and relia- 
bility studies have been completed. Tnter- corre lat ions have been estab- 
lished between various test factors. Test items are analyzed. In short, 
this program, staffed with test specialists and having modern computer 
facilities available, is considered by the project staff as being the 
most promising of all test development, administration, and evaluation 
presently being carried on. 

However, there is still a great variation in the methods and 
procedures employed by the military in developing performance tests at 
the various military centers. They range from careful analysis of the 
items to be evaluated and scientific statistical treatment of the data 
to the personal observation and subjective judgements of examiners who 
rale assigned tasks which are derived from the instruction content. 

Careful job analysis of skills and knowledge required on specific pieces 
of equipment play an important part in establishing examination control. 
Attempts are being' made to reduce subjective judgements. The results of 
the military establishments in measuring occupational competency, and 
their scientific efforts to establish the validity and reliability of 
such measuring instruments and practices, are of substantial value to 
the National Occupational Competency Project. 



o 



45 



PART FI Vi: 



SUMMARY Or OBSERVATIONS AND CONCLUSIONS ON NATION A*. 
EFFORTS AT OCCUPATIONAL COMFETLNCY rKSilNC 



I. OBSERVATIONS OF NATIONAL EFFORTS AT OCCUPATIONAL CONTmLNCV ! C;>T T NC 

Attempts at evaluating achievement or competence are as old as 
the existence of a written language, "There are records of the use of 
written examinations, about 4000 years ago, in the schools o r ancient 
China 11 , In fact, some form of civil service existed during the reign 

of the various dynasties which utilized examinations for the selection 
of civil servants. Among records of ancient Greece have been found 
documents dealing with the treatment of test data, Walden has stated; 

"2300 years ago, in academic subjects in Athens, complex 
explanations of statistical treatment of t«.-st data were 

* O 0 

discussed.’ 1 

A . Major Test Development Lfforts Concent r ate d 
in Academic Subject s 



Present literature contains a wealth of information on contin- 
uing test development and procedures, statistical treatment of test 
results, and the search for better, more accurate, valid and reliable 
instruments. However , these efforts relate primarily to measurement of 
human characteristics, such as: abilities, aptitudes, intelligence, 
emotional stability, occupational interests and related areas. 

Achievement testing has largely been confined to academic 
subjects. The larger number of students involved in such testing 
programs has provided the impetus for greater effort than has been 
possible with occupational competency tests. Studies gave such efforts 
greater priorities. The development of ever- increasing skills in the 
world of work, and the need to develop these skills in others, requires 
greater numbers of teachers and increasing needs for developing instru- 
ments to evaluate occupational competence. 

The search of the literature has shown a sparsity of published 
information on occupational competency testing. Most of the material 
identified was in the form of written pencil and paper tests. There is 
gradual recognition, supported by research, during recent years, that 
the evaluation of occupational competency in highly skilled areas re- 
quires measurement by performance testing. Among the states govern- 
ment agencies, industry and the military contacted, there was a nearly 
unanimous expression of opinion that occupat ional competency perform- 
ance tests are needed for the skilled occupations. Tack of funds, 
•inavailability of adequately trained vocational education testing 
specialists, and limited testing facilities were ci Lid as deterrents 
far i : c f ; test development 
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B . Efforts of Gov e rnment Agencies, Industry 
and the Military 

Civil Service Agencies on the federal, state and local levels 
rely heavily on written examinations. lor some fields, performance 
tests have been developed on a limited scale. A few efforts have been 
made to standardize and validate such instruments. The major reliance 
has been on face validity and the authority of occupational specialists. 
Published data on such tests has not become available. 

The response of industry to the survey requests indicated a 
strong interest in the effort of this project. However, with only a 
few exceptions, industry relies upon supervisory judgement to appraise 
occupational proficiency. No published material dealing with proficiency 
appraisal, through performance tests in industry, was located. It may be 
assumed that the limited number of firms employing such tests wish to 
keep the L esults obtained for internal information only; or, that such 
information is not available. 

In direct contrast, the military has made strong efforts to 
evaluate the proficiency of military personnel, at least, as part of 
the overall competence, through performance tests. While there is much 
variation among the many service schools in the manner in which perform- 
ance proficiency tests are developed, the greatest progress has been made 
in the Army Enlisted Evaluation Program, the Signal Corps Training Schools 
and several Navy Electronics Schools. 

The specific nature of the skills required for the highly 
technical military occupations, the need of preparing the individual 
for effective performance in his assignment, and the ever-increasing 
mechanical and electronic complexity of military equipment and machin- 
ery have given strong impetus to the establishment of special test 
development and control groups which are staffed by test specialists. 

There is an increasing trend to develop, control and maintain written 
and performance tests for certain areas from a central test development 
center. A great deal can be learned from these practices which are now 
carried on. 

C . Efforts Among State Vocational Education Units 

Among the many states, the procedures of evaluating the occu- 
pational competency of tradesmen for teaching in vocational programs 
differs widely. In sixteen (16) states and twenty (20) teacher education 
centers, test development has been concentrated on nine (9) major indus- 
trial occupations areas. States have duplicated each other’s effort. 

The minimum duplication was three ^3) tesLs developed for the same occupa- 
tion; the maximum, twenty {2 O' tests developed by that many institutions 
for the same occupation. The quality of the tests ranged from fairly 
elementary measurement of occupational competency to sophisticated levels. 
Two (2) states have published limited information concerning validity 
and reliability of only a few tests. No such information was available 
from the remaining states* 
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The lumber of states and institutions concerned wiLh the 
recruitment, training, and certification of Leathers of vocational 
education was greater than indicated by earlier surveys. However, 
only Mir l. (3) of the states reporting had assigned individuals full 
time to the development and administration oi occupational proficiency 
Lcs t s . 



Limitations cf funds and lack of qualified personnel have 
been serious constraints to occupational proficiency test development. 
Vet, the survey results indicate much duplication of effort among the 
States and Industrial Teacher Educator Programs. There is a strongly 
expressed concern that occupational proficiency tests are urgent Iv 
needed to establish valid and reliable evidence of prospective 
teacher's occupational competency, which can bo used as part of the 
overall requirements for cert i f ica t ion , admission to vocational teacher 
education program?, and credit towards graduate and undergrac.ua L e 
programs c f study. 

Research evidence supports the position that written tests 
alone are not adequate to establish the level of occupa t ions 1 competency 
needed for teaching in vocational programs. While a limited number of 
states have used written and performance tests for the evaluation of the 
occupational competence, there is growing recognition that quality 
should be developed and made available to all states interested in a 
testing program. 



The Military and, to some degree, the Civil Service agencies, 
have shown the feasibility of using centrally constructed and processed 
tests. 



Although occupational competency tests have been used on a 
limited scale for many years, there is greater urgency coday for 
better, more efficient and more accurate measuring instruments because 
of changing conditions under which craftsmen acquire their skills. 

The findings of the survey of "The State af the Art" point directly to 
the establishment of a national center for ihe development and adminis- 
tration of occupational competency tests. 

Jl. CONCLUSION'S ON HIE STATIC OF fH *•’ ART ON NATIONAL OCCUPATIONAL 

COMPETENCY TESTING EFFORTS 

Ihe State of the Art Study provided crystal clear evidence 

A. lha t not a single professional testing agency, guvernmt. nta 1 
organization, educational body, labor or management group 
was satisfied with cither the qualitative or quantitative 
aspects of occupational canpeLcncy testing; 

R . I hat an exceedingly limited amount of date or experiences 
relating to occupational crtnpetency L?s t development, 
administration, validation, oi research results are recorded 



C. That each organization carrying on significant work in 
occupational testing recognizes that adequate measures 
require testing in both the theory (technology) and the 
skills (performance) of each occupation, that one without 
the other is not valid; 

D. That exceedingly limited numbers of competency tests have 
bnen prepared to measure both skill and technological 
knowledge ; 

E. That each group recognized its limitations and the need 
for considerable refinement in the tests and testing 
program, as well as in the area of supportive research; 

F- That in occupational competency testing the test items are 
related to a grid whose major components are derived from 
a systematic breakdown and job analysis of the manipulative 
tasks to be performed and the accompanying related techni- 
cal knowledge; 

G. That validity and reliability have not been established, 
largely because no vehicle has existed for collectively 
pooling test data, or even using common test instruments 
over a large enough population to be able to analyze and 
synthesize significant findings or experiences; 

H. That only limited numbers of highly experienced and skilled 
occupational competency test construction, administration, 
and research persons in vocational indust rial/ technical 
education and training are engaged full time in this effort 

I. That ro single unit, up to this point, has had a combina- 
tion of sufficient funds, facilities, equipment, personnel, 
or great enough populations of test participants in one 
occupational field, to be able to adequately develop, 
administer or evaluate the results; 

J. That the development of complete occupational competency 
examinations requires substantial investments of human and 
material resources, significant number s taking an examina- 
tion, witn the results pooled for '.statistical analysis and 
other treatment of the data in escablishing norms, and 
other information that may be studied in depth; 

That ther-e is excessive unplanned duplication of effort in 
a restricted number of occupational fields; 

L. That there ir a serious lack of dcata and research activity 
in occupational competency testing; 

M. That there is reed for nationally pooling of resources and 
experiences to establish reliaoility, ard validity in 
economical and efficient ways; 
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N. That many forms of tests for a single occupational field 
are needed to serve many different measurement functions 
for different parts of the occupation and different combi- 
nations; 

O. That there is no system of depositories or central store 

of occupational competency test data, test items, test banks ; 

P. That in vocational education a Consortium of States effort 
to develop, administer and validate tests requires the 
investment of significant sums for, at least, twenty (20) 
major industrial occupations* the employment of competent 
staff on a full-time basis, supportive "software" and - 
"hardware", and a cooperative arrangement for feedback 
upon which judgements may be made on the effectiveness of 
each instrument and the value of test results and testing 
program . 

Concurrent with the State of the Art Study and, as part ot the 
project, interested states were brought together to explore Lhe need 
for and feasibility of establishing a National Consort ium on Occupa- 
tional Competency Testing. Kxcerpts from this report on The State 01 
the Art were used as a foundation for reporting conditions as they exist, 
and the recotmendat ions which were made by the many concerned groups. 

Thus, a final conclusion may be added, to the effect ihut 
practically all of the states are interested in a vocational / industrial 
technical occupational competency program and would participate in a 
national effort. The organi zat ional structure and management functions 
of this activity is reported as another part of the overall project. 

At this time, the whole national effort in occupational eompi tency 
testing is at a significant breakthrough poinL, 
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APPENDIX C 



SELECTED SUPPLEMENTAL RESOURCE MATERIALS 



Selected resource materials on measurement principles, practices and 
actual test instruments, developed by and for the military, may be 
secured from the United States Department of Commerce, National bureau 
of Standards, Clearinghouse for Scientific and Technical Information. 
Springfield, Virginia, 22151* 



Achievement Tests, Air Force Personnel 

Airman qualifying examinat ion- -66 administered as a confidence 
tost . 

AD-677 219 5J U69 01 HCS 3.00 HFS 0.65 

Achievement Teats, Correlation Techniques 

A comparison of four models for making predictions across 
inst itu t ions . 

AD-673 5J U68 21 HCS 3.00 MFS 0.65 

Achievement Tests, Mathematical Analysis 

Use of army school samples in estimating AC 8 test validity 
AD-677 388 5J U69 01 HCS 3. 00 MFS 0.65 

Achievement Tests, Supervisory Personnel 

A study of the effects of examiner race, 
test responses of negro examinees 
PB-178 901 5J U68 18 HCS 

Achievement Tests, Army Personnel 

Measurement of achievement motivation in Army security 
agency foreign language candidates. 

AD-623 0-1 U65 23 

Achievement Tests, Electronic Technician 

Development of an electrical electronic test battery 
AD-612 891 U65 09 HCS 3.00 

Achievement Tests, Performance (Human) 

Confidence, ccrrectness, and difficulty with non-psychophyoicu i 

comparative judgements 

AD-611 979 U65 08 

Achievement Tests, Programmed Instruction 

Measures on ability and programmed instruction performance 
AD-6 29 443 5H U6u OS HCS 3. CO MFS 0.6 5 

Achievement Tests, Psychometrics 

Difficulty and other correlates of criticalness response 
AD-600 702 U HCS 3.00 



sex, and style on 
3.00 HFS 0.65 
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Acuievement lefts, Test Construction (Psychology) 

Development and preliminary evaluation of the United States Navy 

automotive knowledge test 

AD-627 844 5H U66 06 

Aptitude Tests, Air Force Training 

Selection of Foreign Students for training in the United 
States Air Forco 

AD-683 725 51 U69 09 HCS 3.00 HFS 0.65 

Aptitude Tests, Creativity 

The predictive validity of certain measures of creativity in 
machine design 

AD-662 696 5J U68 03 HCS 3.00 HFS 0.65 

Aptitude Tests, Electronic Technicians 

The effect of electronic aptitude on performance of 
procedura 1 ized troubleshooting tasks 

AD-664 889 51 U68 07 KCS 3.00 ii- S 0.65 

Aptitude Tests. Factor Arilysis 

A facto* analysis of the symbolic-eva luat ion abilities 
studies of aptitudes of high-level personnel 
AD-464 259 3J Uf 8 19 MCS 3.00 



Aptitude Tests, Maintenance Personnel 

Effect of Ambiguous test results on troubleshooting performance 
AD-661 609 U68 07 MCS 3.00 KFS 0.65 



Aptitude Tests, Programmers 

Evaluation of Tests for improvement of programmer trainee 
selection 

AD-661 609 51 U68 02 HCS 3.00 KFS 0.65 

Aptitude Tesis, Air Force Personnel 

Prediction by career field of first term airman performance 
from selection and basic training variables. 

AD-600 781 U HCS 3.00 



Performance on Airman Qualifying Examination by Regional areas and by 
sex 

AD-617 335 U65 16 HCS 3.00 

Aptitude Tests, Manpower Studies 

Army School Training Perfornance of Da Scoring Low on Afot, 
AD-609 045 U HCS 3.00 



Investigation of the United States Navy Clerical Aptitude Test, 
Form 6 , 

AD-620 680 51 L66 20 HCS 3.00 HFS 0.65 

Aptitude Tests, Personnel Management 

Job performance of EM Scoring Low on Afot 
AD-602 980 U HCS 
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3.00 



Development of Core Training for F-86D Electronic 

Maintenance positions III Derivation of training characteristics 
AD-669 399 51 U68 14 HCS 3.00 NFS 0.65 

Development of Core Training for F-86-D Electronic 

Maintenance Positions IV Principles and Techniques 
AD-669 400 51 U68 14 HCS 3.00 MFS 0.65 

A Performance-Oriented Electronics Technician Training 

Program IV Fleet follow-up evaluation of graduates of all 
c lasses 

AD-66 565 51 U68 09 HCS 3.00 MFS 0.65 

Electronic Technicians, Performance Tests 

An analysis of structure and errors in corrective maintenance 
work 

AD-666 568 51 U68 09 HCS 3.00 HFS 0.65 

Electronic Technicians, Performance Human 

A field survey of electronic maintenance technical data 
AD-666 990 51 U68 10 HCS 3.00 HFS 0.65 

The Assessment of Electronics Corrective Maintenance 
Performance II Performance cn the AN/SPS-40 by 
electronics technicians 

AD-673 246 51 U68 19 HCS 3.00 HFS 0.65 

Development of Core Training for F-86D Electronic 

Maintenance positions II Identification of training 
contept 

AD-669 398 51 U68 14 HCS 3.00 HFS 0.65 

Electronics, China 

Communist Chinese Scientific Abstracts' Electronics and 
Electrical Engineering No. 92 

JPRS-43802 9C U68 07 HCS 3.00 HFS 0.65 

Electronics, Eastern Europe 

East European Scientific Abstracts: Electronics and 

Electrical Engineering No. 40 

JPRS-4 324 2 9C U68 01 HCS 3.00 HFS 0.65 

East European Scientific Abstracts: Electronics and 

Electrical Engineering No. 41 

JPRS-43492 9C U68 05 HCS 3.00 HFS 0.65 

Engineering Personnel, Selection 

Qualification Examinations urged for Plant Engineers 

Performance Decrement 

? El 59 Miniature integrating GYRO Sterilisation exposure studies 
at 300 deg final report 

M66- 10616 <4B S04 01 HCS 3.00 HFS 0.65 
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Forecasts and Appraisals for Management Evaluation Volume 2 

M66-29966 9B S04 16 HCS 3.00 HFS 0.63 

Forecasts and Appraisals for Management Evaluation Volume 2 

M66-29967 S04 16 HCS 3.00 HFS 0.63 

Performance Tests 

Individual Task Performance in a changing social structure 
AD-601 868 U HCS 3.00 

Performance Tests, Air Force Personnel 

Describing aiiman performance in the Administrative Career 

ladder by identifying patterns of trait ratings 

AD-653 544 51 U67 16 HCS 3.00 HFS 0.65 

Performance Tests, Air Traffic Controllers 

problems in air traffic management IV comparison of pre- 
employment, job related experience with aptitude tests as 
predictors of training and job performance of air traffic 
AD-603 416 U HCS 3.00 

Performance Tests, Army Personnel 

Job performance of HM Scoring low on Afot 
AD-602 980 U HCS 3.00 

Performance Tests, Correlation Techniques 

Comparison of Random pairs and real pairs on a simple 
auditory counting task 

AJ-638 306 51 U66 20 HCS 3.00 HFS 0.65 

Performance Tests, Electronic Technicians 

The development and application of absolute scales of 
electronic job performance 

AD-602 392 U HCS 3.00 

Performance Test6, Naval Personnel 

Post -Training performance criterion development and application 

Performance Tests, Reliability 

Relation of job qualification ratings to performance ratings 

of basic training instructors 

AD-607 670 U HCS 3.00 

Performance Tests, Statistical Data 

Studies in research methodology VI the central limit effect for 
a variety of populations and the robustness of Z, T, and F 
AD-612 886 U65 09 HCS 3.00 

Performance Tests, Submarine Personnel 

Research on the development of shipboard performance measuras 
and performance judgements 

AD-610 489 U6f' 06 HCS 3.00 



Performance Tests, Time Studies 

Operator Performance Studies: II, Learning analysis from 

three-plane motions 

AD-603 528 U HCS 3.00 

Performance Tests, Transfer of Training 

A replication of paired-associate learning as a function of 
S-R similarity. 
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APPENDIX D 



ORGANIZATIONS AND INSTITUTIONS CONTACTED 
Organizat i on e in Vo cational Edu cation 

State Directors of Vocational Education or their equivalent in each 
State, District of Columbis, Virgin Islands, Puetto Rico. 

National Association of State Directors of Vocational Education 
(A1 1 Members) . 

National Association of Industrial and Technical Teacher Educators 
(All Members) . 

National Association of State Supervisors of Trade and Industrial 
Education (All Members). 

Research Committee of the American Vocational Association, 

Regions I to VIII . 

State Vocational Research Centers and Bureaus. 

National Vocational Research Center - Ohio State University. 
National Vocational Research Center - University of Wisconsin. 



G overnment Agencies 

United States Department of Commerce 
United States DepcrtraenL of Labor 

United States Department of Labor - Office of Manpower Automation 
and Training. 

Equal Employment Opportunity Commission 
United States Civil Service Commission 
New York State Civil Service Commission 

New York State Department of Labor - Internal Training Division 

Apprentice Training Division 
Manpower Training Division 

New York City Civil Service Commission - Professional and Technical 

Evaluat on and Testing 
Division 
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Professional Organizations 



American Psychological Association, Washington, D. C. 

American Educational Research Association, Washington, D. C. 
National Council for Measurement in Education, Washington, D. C 
Military Testing Association, Governor’s Island, New York, New York 



Private Organizations 

American Institute of Certified Public Accountants 
Mew York City, New York 

Associated Personnel Technicians 
Wichita, Kansas 

Educational Testing Service (visited) 

Princeton, New Jersey 

Industrial Psychology, Inc. 

New York, Now York 

McAnn Associates 
Philadelphia, Pennsylvania 

National League for Nu sing 
New York City, New York 

Psychological Institute 
Lake Alfred, Florida 

The Psychological Corporation (visited) 

New York City, New York 

The Psychological Research Corporation 
New York City, New York 

Psychological Services, Inc. 

Los Angeles, California 

Science Research Association, Inc. 

Chicago, Illiroi3 

American Transit Association 
New York City, New York 



Mi 1 itary 
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United States Air Force 
Lackland Air force Base, Texas 



United States Air Force (visited) 

Wright Patterson Air Force Base, Dayton, Ohio 

United States Air Force 

Lowry Air Force Base, Denver, Colorado 

United States Air Force 

Personnel & Training, Research Center 

San Antonio, Texas 

United States Army Ordnance School (visited) 

Aberdeen, Maryland 

United States Army Signal Corps Training School (visited) 
Fort Monmouth, New Jersey 

United States Army Enlisted Evaluation Center (visited) 
Indianapolis, Indiana 

United States Army Armor Training Center 
Fort Knox, Kentucky 

United States Coast Guard (visited) 

United States Coast Guard Training Center 
Governor's Island, New York 

United States Navy Electronics Laboratory 
San Diego, California 

United States Navy (visited) 

Great Lake? Naval Training Center 
Electronics Schools 
Great Lakes, Illinois 



T rade 6 Industrial Organizations 



American Association of Industrial Management (visited) 
New York, New York 

American Management Association (visited) 

New York, New York 

Associated Electrical Industries 
New York, New York 

National Association of Manufacturers (visited) 

New York, New York 

National Metal Trades Association 
New York, New York 
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National Tool, Die and Precision Machining Association 
New York, New York 



(visited) 



Industrial Companies 

A. C. Electronics Division of General Motors 
Public Relations Department 
Milwaukee, Wisconsin 53201 

Air Reduction Company, Inc. (visited) 
Personnel Department 
New York, New York 

American Can Company 
New York, New York 

American Car and Foundry (visited) 

Division A.C.F. Industries, Inc. 

New York, New Yotk 

American Cyanamid Company 
Personnel Division 
Wayne, New Jersey 

American Telephone and Telegraph Company 
New York, New York 

Armco Steel Corporation 
New York, New York 

Bethlehem Steel Corporation (visited) 
General Offices, New fork, New York 

Buirk Motor Division 
Service Training Department 
Flint, Michigan 48550 

Cadillac Motor Car Division - General Motors 
Service Training Department 
Detroit, Michigan 48232 

Delco-Moraine Division - General Motors 
Personnel Department 
Dayton, Ohio 45401 

Delco Radio Division - General Motors 
Public Relations Department 
Kokomo, Indiana 45901 

E. I, du Pont de Nemours and Company 
Wilmington, Delaware 
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Foote Mineral Compan 
Exton, Pennsylvania 

General Electric, (visited) 

New York, New York 

General Dynamics Company (visited) 

General Education Services 
New York, New York 

General Motors Corporation 
Public Relations 
Detroit, Michigan 

Inland Steel Products Company 
Milwaukee, Wisconsin 

International Business Corporation 
White Plains, New York 

International Mineral and Chemical Corporation 
Stokie, Illinois 

Mead, Johnson and Company 
Evansville, Indiana 

Minnesota Mining and Manufacturing Company 
St. Paul, Minnesota 

Owen-I llinois Glass Company 
Toledo, Ohio 

Spartan, 

Division of Spartan Corporation 
Jackson, Michigan 

Visking Company 

Division of Union Carbide Company 
Chicago, Illinois 

Western Electric Company (visited) 
Psychological Testing Department 
New York, New York 

Western Electric Company - Personnel (visited) 
New York, New Ytrk 

Westinghouse Electric Corporation (visited) 

New York, New York 
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Labor 



United Auto Workers 
8000 East Jefferson Avenue 
Detroit, Michigan 48214 

International Association of Machinists & Aerospace Workers 
1300 Connecticut Avenue, N. W. 

Wajhington, D. C. 20036 

National Joint Steamf itter-Pipe Fitter Apprenticeship Committee 

U. A. Building 

901 Massachusetts Avenue 

Washington, D. C. 20001 

National Joint Apprenticeship and Training Committee 
for the Electrical Industry 
1730 Rhode Island Avenue 
Washington, D . C. 

United Brotherhood of Carpenters and Joiners of America 
Apprentice and Training Department 
76 Lamdka Avenue 
Staten Island, New York 10308 
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